Integrate Vyos with standalone web filtering device?

You would have to forward traffic to your device. Preferably it handles all types of traffic. Otherwise you can forward dport 443 towards a specific IP.

And if you have any other known https destinations with different port numbers - redirect corresponding traffic explicitly.

Thanks both.
As suggested, is there a way to check the device is live and then forward traffic or do a fallback to another device.

You can use policy routing to match HTTP and HTTPS traffic and point it at a next-hop that is an external transparent proxy.

The way Quagga works if the interface goes down (e.g. is disconnected) it will "fail open" and traffic will go out the normal default gateway.

Short answer: not really.

You could possibly abuse the wan-load-balancing facility I think with some additional clever NAT rules. Your proxy would become a default GW. You should use a different testing-target then ICMP. There's an example/script for HTTP requests in the code. I'm not sure whether that's documented somewhere.

@EwaldvanGeffen have you given the method I described a try on VyOS? I know it works on EdgeOS and pre- 6.4 releases of Vyatta and honestly haven't tested it on VyOS because it's not something I have a need for... so it very well could work differently/be broken on VyOS, but that would be surprising.

Example:

https://help.ubnt.com/hc/en-us/articles/204962204-EdgeMAX-Policy-based-routing-for-transparent-proxy

It should work for VyOS just changing it from "firewall modify" to the policy-route syntax.

Looking back, I actually posted exactly this example on the Wiki in one of my updates:

http://wiki.vyos.net/wiki/User_Guide#Policy_Routing

Note that in this configuration the Squid server must be configured to have forwarding enabled (e.g. as if it were a transparent proxy).

or do a fallback to another device.

@rps I'm not the topic starter ;) Your solution would work but not do any dynamic probing regarding availabilty. I know PBR has no option to make dynamic decisions and WLB does.

In theory, you could have the web filter be a pair of servers using VRRP.

I think @hmkias was mainly looking to see if there was a way of supporting an external transparent proxy, and with policy routing that should be doable. In terms of the redundancy and failover, that's a more complicated discussion, I agree.

It seems like the behavior that's being looked for is something similar to WCCP, which is a Cisco protocol that allows for traffic to be sent to a collection of cache servers with load balancing (round-robin, I think) and service availability checks; basically using GRE tunneling. The only implimentation for WCCP I'm aware of is wccpd, which doesn't seem to have seen active development for 15 years, so it likely would take a bit of work to pull in to VyOS.

http://wccpd.sourceforge.net/
https://en.wikipedia.org/wiki/Web_Cache_Communication_Protocol

Thanks @rps, @EwaldvanGeffen.

I would start with PBR in the first phase for supporting the external proxy.

BTW: Would it possible to patch squid in Vyos?

@hmkias Patch Squid for what?

Patch for HTTPs filtering.
I have to remove the existing version and install a new patched version from source.

Would it be possible?

@rps I think he needs a more modern version of squid with sslbump support. I wouldn't put any effort in WCCP, it seems fairly legacy to me.

I m thinking on two approaches to the problem, WCCP or patching Squid. Ultimately the complexity and time decides the way.

@hmkias I think that some kind of a daemon would be required to "coordinate" between the squid machine to the VYOS.
I had an idea about it in the past but never had the chance to actually implement it with vyatta.
However I have seen that in ZEROSHELL there is a very nice feature which test for proxy IP level availability.
How complex would it be to make a condition to the policy based on a lock file?

@EwaldvanGeffen technically we can simplify it into a form of a script that monitors the service using http or another tcp\udp based and would flag the avaliability of the service.
The marking and forwarding rule can be automativally bypassed if the service is flagged as down.
Anyone interested working with me on this?
It's basically a simple conditional PBR.. and since WCCP is "OK" for tiny routers for beafy machines such VYOS have I believe that it would be a piece of cake to cook this up.

@elico it's pretty simple since WLB supports custom tests for gateway/targets. You can simply script it up to that.

@EwaldvanGeffen Can you help with giving an example of implementing this?
Like with a tiny ping that returns a status code?
(I do not know what WLB is...)

Wan-load-balance. Example is here: https://github.com/vyos/vyatta-wanloadbalance/blob/current/scripts/http_test.pl and implementation https://github.com/vyos/vyatta-wanloadbalance/blob/current/templates/load-balancing/wan/interface-health/node.tag/test/node.tag/type/node.def

@EwaldvanGeffen WLB has a difference from PBR and what is required a PBR.
The code is not something I was looking for but an example of implementation in the configuration.
Then I will be able to look at the code and understand what might be applied to PBR compared to WLB.

@elico if you apply a 'source my-lan-clients, destination port-80, proto tcp' rule with gateway your proxy server + the custom testing-target script. If the proxy is up it will be routed towards it. If the target goes down, without any other policies the packet will fall onto PBR and then routing. Isn't that the behaviour you were looking for?

@EwaldvanGeffen apply this rule on what? a WLB?
the WLB from what I understood required an interface per gateway while PBR allows me to route the traffic towards any of the gateways which can be the next-hop ie 10.0.0.100/24 or 10.0.0.101/24.
This is what I remember from vyatta and I haven't digged into the subject since I have a huge gap ahead as far as I can see.

Mentioning: http://pastebin.com/yZLVRfnA
Which is an example of how would WLB work with a custom script.

And adding a little explanation on WCCP as a base to a PBR "master" daemon which will be able to monitor the proxies and to flag them as up or down.
Based on the daemon mark another script will run let say every 15 second a test loop and will execute couple commands(maybe using the python library https://github.com/vyos/python-vyos-mgmt).

WCCP:
The WCCP protocol is being used between the router and the caching\filtering proxy or router for the next task:
service registration ie announce on the service capabilities and interception method used(L2 or L3 over a special GRE tunnel)
service state verification and validation(udp ping pong)

All of the other features which are connected to the WCCP clustering features are configurable on the Cisco device and not the proxy while it would be pretty simple to add into the WCCP announcement something like weigh as a part of the registration.

The use of UDP in WCCP is to lower the CPU requirements and complexity of the software handling these.
Since VYOS currently runs only on x86 and x86_64 systems and couple cycles of CPU can be spared we can move on from trying to support WCCP and use some other software that will communicate with the VYOS or another Linux x86 based route server over some UDP or TCP connection that will result eventually in a PBR addition or removal from the VYOS router.