I've written a simple, step-by-step tutorial on getting a web caching proxy (squid in particular) working on a desktop machine running Windows. An excerpt is below:

So what is squid or a caching proxy in general? The squid website says:

"Squid is a caching proxy for the Web supporting HTTP, HTTPS, FTP, and more. It reduces bandwidth and improves response times by caching and reusing frequently-requested web pages."

So why would you want to use it? From my experience, if you are in any of the following situations:

• You want your browsing to be snappy in general (helps on cable/ADSL too!)


• You are shaped and want better performance when browsing your frequently-visited sites


• You are about to get shaped and want to reduce your quota consumption


• You are on a flaky, high-latency or slow connection (e.g. laptop Wi-Fi, wireless broadband, dial-up)

Read the rest of the tutorial: Link

If it's for a LAN I could understand it's use, but surely the browser caches are suitable enough for a single machine ?

cool, i need to test against different proxys. does it run as a service?

but surely the browser caches are suitable enough for a single machine ?

^ You might want to read the rest of the tutorial, I discussed that briefly :)

Basically the proxy's DNS caching helps heaps (of course you could DNS cache other ways if you wanted to), plus catching what the browser hasn't cached. I installed squid when I was in Sydney roaming on a wireless broadband modem and it really made browsing much less painful.

does it run as a service?

The tutorial tells you how to, yes.

last edited by parabol at 19:16:57 26/Jun/08

cool. ill set it up today, and run a few users against it. see how it goes.

i run dnscache under linux and it rocks.

i do run squid under linux and make squid use my local dnscache, it works lots better then squid alone.

i set up squid using 2 instances of squid and dansguardian for a couple of clients

integrates into AD, does content filtering and proxying, awesome shizzles

yeah, it is pretty neat. i recently set up squid as a reverse proxy to take load off an extremely busy web server.

sustained throughput from the cache was 90meg/s

Sweet, I'm gonna try this when I get home. Thanks for the tute.

parabol when you come to pimping Squid SNMP monitoring via Cacti then drop me a line and I'll help out, here are some sexy things it can generate;

http://www.gronks.com/qgl/squid1.png
http://www.gronks.com/qgl/squid2.png
http://www.gronks.com/qgl/squid3.png

sexy graphs.

got an export of your templates?

Linky here for the data query templates, can't remember if they auto-generate the graph templates or not but if it doesn't let me know and I'll export them.

You have to forward the OIDs for squid through to the squid SNMP service, net-snmp or windows SNMP won't pick it up. With Windows SNMP it's a bit tricky since you can't do forwarding so I had to install net-snmp which forwarded the squid OIDs to squid and the other OIDs to the Windows SNMP service which was running on a different port.

last edited by TicMan at 10:37:51 27/Jun/08

Hahah this rocks, i hope it helps with facebook on thre missus's pc while im gaming :)

Setup lan proxy so when viewing vids it should help a heap

TicMan, do you have anything that parses squid logs for usage analysis/reporting? I was looking at a few things and there's a bunch of different options (many of which look great), just wondering if you (or anyone else) have any recommendations. Calamaris and SARG I think were the 2 main ones I was looking at.

Calamaris was the one I was looking into but never got around to implementing it.. there's a limited selection of opensource squid analyzers that produce nice reports with graphs and "Top 10" lists, etc.

Trog, I looked at a few options, and found almost all of them to be lacking in some way or another. I eventually settled on Mysar. It saves the data to mysql, so I actually pull most of my reports directly from the database.

Allowed me to do some cool stuff like show users how much data they've used today on the intranet so they can be more aware of the impact their browsing has etc.

I've just installed your templates ticman, they look pretty sweet.

ah awesome! ive been waiting for this. thanks heaps fellas

Allowed me to do some cool stuff like show users how much data they've used today on the intranet so they can be more aware of the impact their browsing has etc.

cool that's pretty much what I wanted to do as well. I'll check out Mysar, thanks.

Rad, I just set this up, thanks for the straight forward tutorial, now lets see how many horrors of the internet it can hold with a 1GB cache!

happy to share php code to do it when you've got mysar installed.

digg click fiends, I just digg-ified this news post

happy to share php code to do it when you've got mysar installed.

sweet dude ta

Where do you set the disk cache size in the squid.conf what line am i searching for?

The variable is cache_dir , I set it to something like:

cache_dir ufs c:/squid/var/cache 1024 16 256

(1024MB max swap)

EDIT: you might also want to change maximum_object_size to make sure big-ish stuff gets cached.

last edited by parabol at 15:07:33 27/Jun/08

cache_dir it is the same setting for location.

Cheers guys :D

used SARG at work

ha dammit i setup squid on my windows machine about 2 weeks ago.

3. Run: squid -z <<<< boy did that give me nightmares, found 1 obscure mention of it on a site.
so Props to that in the tutorial, lots i found didnt mention that at all.

I used sarg at work but for some reason one random day it stopped collecting any results ... no idea why :( this was about 2 months ago

Cool guide :)

Hmmm, anyway to speed up downloads if you're using another system as your Squid proxy? Downloading files larger than about 3MB has suddenly been crippled (around 40-50KB/s vs. 500KB/s+ before.

Nm...seems to have come right on its own.

last edited by Twisted at 19:23:21 30/Jun/08

I've just set this up, very easy. Few questions. 1. I have an interal site running on 1024. The proxy doesnt like it, complains about authentication. The site needs AD authent. Any ideas?

2. Ive got a list of blocked sites. How do I import the ips?

Few questions. 1. I have an interal site running on 1024. The proxy doesnt like it, complains about authentication. The site needs AD authent. Any ideas?

I have no idea what all that means, but does this help?

http://www.mail-archive.com/squid-users@squid-cache.org/msg43483.html

2. Ive got a list of blocked sites. How do I import the ips?

I've never done it, like this?

http://www.debian-administration.org/articles/399

cool. i found in the squid.conf a list of allowed ports. added 1024 and it fixed that.

having issues with the blocked sites, prolly not finding ths blocked list file.

Does anyone know of a good content filtering software which goes well with squid?????

^ could chain it to Privoxy?

Privoxy is a non-caching web proxy with advanced filtering capabilities for enhancing privacy, modifying web page data, managing HTTP cookies, controlling access, and removing ads, banners, pop-ups and other obnoxious Internet junk. Privoxy has a flexible configuration and can be customized to suit individual needs and tastes. Privoxy has application for both stand-alone systems and multi-user networks.

http://www.privoxy.org/

(the FAQ is very informative)

how has this effected players pings (if at all)?

ah k it's just setup for the web browser

Squid in SOHO environment appears to be of neglible benefit because of dynamic content hosting.

Explain.

Squid in SOHO environment appears to be of neglible benefit because of dynamic content hosting.

I created some things in squid.conf which has helped a lot with Dynamic sites:

acl dynamic_sites dstdomain .youtube.com
acl dynamic_sites dstdomain .facebook.com
acl dynamic_sites dstdomain .ausgamers.com
acl dynamic_sites dstdomain .google.com

cache allow dynamic_sites

I put that before hierarchy_stoplist cgi-bin ? in the config file. Getting a lot more cache hits now.

About 20%-25% of hits to sites like Google and Facebook, etc now use the cache. I think by default Squid won't cache any dynamic site stuff at all (not even the static images and stuff on dynamic sites).

Thanks Twisted giving that a shot now

I'm getting a lot of "The requested URL could not be retrieved" and "Invalid Response" error pages lately.

It just started happening out of the blue. Dell.com , yahoo.com and various other sites are unviewable until I put them on the proxy bypass list :/

run wireshark or something and look at what squid is sending/receiving

i wanna know, does this tutorial used on windows server 2003 and how to used interface like SARG.

can u help me...

for ur attention, thank's so much

Thanks for showing how it's done, I have a question set it up and use our firewall to point to the squib proxy, everything is working while browsing the website on the client terminal but when we go to login Yahoo mail or hotmail mail it comes up can't login with the screen full off errors by squid, is there anything I am suppose to do in the config to solve this?

I had a prob running this:

squid -i -n squid -O "-D"

Error message is:

OpenSCManager failed

Trying under Vista 32

**EDIT** btw, I cut and pasted the line from tute, so no syntax error with that letter O being replaced by zero or whatever

**EDIT x2** the problem was UAC. I put the following lines in batch file and ran as admin

cd C:\squid\sbin
squid -i -n squid -O "-D"
pause

UAC needs to go into the Qgl blamegen

This the the errors I am getting when I access Yahoo.com mail after login.

ERROR
The requested URL could not be retrieved

--------------------------------------------------------------------------------

While trying to process the request:

GET /echarts?s=%5EHSI HTTP/1.1
Accept: */*
Referer: http://finance.yahoo.com/q?s=%5EHSI
Accept-Language: en-us
UA-CPU: x86
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; T312461; Q312461; User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://bsalsa.com) ; InfoPath.1; .NET CLR 2.0.50727; .NET CLR 1.1.4322)
Host: finance.yahoo.com
Connection: Keep-Alive
Cookie: B=34gdhcl380mj5&b=4&s=6u; SO=v=0.4&t=1220840008; YLS=v=1&p=0&n=0; F=a=DHGd26U[snip]; Q=q1=AACAAAAAAAAAAA--&q2=RtaGOg--; U=mt=_mNT5p2MhYr.aBYuRy1Q8AX5y55Zc2xTk8vC&ux=TgLTHB&un=fnurouvvt0pnc; LYC=l_v=0&l_lv=10&l_s=q40r3wyrzuztuy5u0uxwyzu103wyt4t4&l_um=0_0_1_0_0; C=mg=1; DNR=1; cna=NlNrADAIvVoBAQbxPgrzDsQ9; ystat_cn_bc=1892624151692278399; PH=fn=YlJNTyAhO8uRgXmw&l=en-US; YSC=0; BA=ba=14227&ip=202.82.16.59&t=1231906832; Y=v=1&n=0s9rn9v9q0e5c&l=a8m8_7adp/o&p=m24vvhk413000300&iz=&r=3v&lg=en-US&intl=us&np=1; T=z=GHFb[snip]; RT=s=1231906851103&u=&r=http%3A//www.yahoo.com/; PRF=cd=symbol%3A%5Ehsi_%40range%3A1y_%40indicator[snip]; TT=tick1=0&tick2=0&tick3=1


The following error was encountered:

Invalid Request
Some aspect of the HTTP Request is invalid. Possible problems:

Missing or unknown request method
Missing URL
Missing HTTP Identifier (HTTP/1.0)
Request is too large
Content-Length missing for POST or PUT requests
Illegal character in hostname; underscores are not allowed
Your cache administrator is webmaster.

Can anyone help please?

Thanks

last edited by Jim at 08:49:26 15/Aug/09

If you're needing to pull data for reports of either webstats or squid logs (anything that drops into a mysql db really) have a look at Prism by SiSense

Hi

Yahoo.com mail seems to went ok now, but there is still something wrong, when I login to hotmail I get this:

ERROR
The requested URL could not be retrieved

While trying to process the request:

GET /default.aspx?n=270930736&wa=wsignin1.0 HTTP/1.1
Host: mail.live.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Cookie: wlidperf=throughput=7&latency=442&FR=L&ST=1232346993754; PPLState=1; RPSTAuth=EwDYARAnAAA[snip]; MSNPPAuth=B6KPFLgn3FEbiZ2[snip]; MH=MSFT; NAP=V=1.8&E=78f&C=gjp[snip]; ANON=A=208E92978701E5887161FB82FFFFFFFF&E=7e9&W=1
Cache-Control: max-age=0



The following error was encountered:

* Invalid Request

Some aspect of the HTTP Request is invalid. Possible problems:

* Missing or unknown request method
* Missing URL
* Missing HTTP Identifier (HTTP/1.0)
* Request is too large
* Content-Length missing for POST or PUT requests
* Illegal character in hostname; underscores are not allowed

ANd when I reply to some sites I get :

ERROR
The requested URL could not be retrieved

While trying to process the request:

POST /my/message_process.asp HTTP/1.1
Host: hongkong.asiaxpat.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://hongkong.asiaxpat.com/my/message_post.asp?reply=17283037
Cookie: ASPSESSIONIDCASAQRST=HLCICMACJLJNOEOJPLABCGHL; __utma=120978705.1921639512.1232351733.1232351733.1232351733.1; __utmb=120978705; __utmc=120978705; __utmz=120978705.1232351733.1.1.utmccn=(direct)|utmcsr=(direct)|utmcmd=(none); AsiaXPAT=UserID=522710&AuthID=FA119C0D%2DCF0C%2D4846%2D9192%2DEE38776DDB36&save=True
Content-Type: multipart/form-data; boundary=---------------------------265001916915724
Content-Length: 1907

-----------------------------265001916915724
Content-Disposition: form-data; name="range"

S
-----------------------------265001916915724
Content-Disposition: form-data; name="users"

saadkhan
-----------------------------265001916915724
Content-Disposition: form-data; name="title"

re:iphone 2G 8G
-----------------------------265001916915724
Content-Disposition: form-data; name="message"

The best price I can do for you is $2600 because the shop will give me $2500 for the full set.


-----------------
January 19

The following error was encountered:

* Invalid Request

Some aspect of the HTTP Request is invalid. Possible problems:

* Missing or unknown request method
* Missing URL
* Missing HTTP Identifier (HTTP/1.0)
* Request is too large
* Content-Length missing for POST or PUT requests
* Illegal character in hostname; underscores are not allowed

Your cache administrator is webmaster.
Generated Mon, 19 Jan 2009 08:06:03 GMT by localhost

The proxy is on the DMZ zone and the firewall is pointed to squid proxy on the DMZ zone. Everthing is let out so I don't know why we get this errors? Can anybody help?

Thanks


last edited by Jim at 08:49:50 15/Aug/09

I installed it the day this post was made and I've been having probs ever since and now I am just back to normal again.

My problem is very simple - Squid was giving me DNS resolution errors erratically for common domains like google.com and hotmail.com

They were absolutely erratic, no pattern at all. Obviously these domains are well-used and would therefore be in the cache - I can't understand why Squid can find them sometimes and not others.

Also what I have found is that Squid doesn't seem to like much pressure. I use SnapLinks addon for Firefox which allows you to open multiple links by selecting them with a right-click and drag rectangle. If I open more than 20 links (an amount that Firefox handles fine) then Squid seems to have a proxy-meltdown.

Anyway, that's my experience, maybe it will send some dude one way or another.

**EDIT** FYI, when I deleted my squid dir it was 110MB - so presumably caching a reasonable amount which is good.

Can anybody help?

^ I already mentioned a few posts back about problems with yahoo/dell, etc.

Haven't looked into it since putting them on the proxy bypass list.

Thanks, I fix it by using http_port 3128 transparent

hi Ticman

can u help me with plotting graphs with cacti plss !! i need to find out hw much bandwidth is getting saved using this squid option.

Thanks for ur help!

(Pre proxy usage) - (Proxy usage now) = savings!

Or add those templates to Cacti, allow SNMP in Squid and connect the two together. I don't have access to a working Cacti or Squid environment anymore so can't offer too much more than that.

Is there an easy way to use Squid to cache Steam and Windows updates? We've got a box on our home network acting as a squid proxy for the other 3 PCs on the network, and we all run Windows and Steam. It'd be nice to not have to download all of these updates 3 times.

I was having problems with the DNS servers, Was anyone eles? But I have managed to fix it and this is how i did it...

Find out your gateway IP/DNS Servers on your PC
go to start > run > type in cmd and hit enter
type in ipconfig
and it will bring a list up like
http://i32.tinypic.com/25ggz1j.jpg

Then add this to squid.confg
http://i30.tinypic.com/2nh21ie.jpg
And save.

go into start > run > type cmd and hit enter
type in net stop squid
type in net start squid
then it should work :)

Is there an easy way to use Squid to cache Steam and Windows updates? We've got a box on our home network acting as a squid proxy for the other 3 PCs on the network, and we all run Windows and Steam. It'd be nice to not have to download all of these updates 3 times.

Yes just make sure is forward to port 3128 as squid's default

1st. Click Start, click Run, type cmd, and then click OK
2nd. Type proxycfg -p localhost:3128, and then press ENTER

Note: Replace "localhost" to your squid server so for e.g 192.168.1.42:3128

How much will latency drop in games? We've gone from 60gb to 30gb and we're struggling. Been want to setup a server with squid for some time now.

most likely, your latency in games won't drop at all because they don't use cached http objects very much ;)

if your web browsing is so heavy that it completely saturates your connection constantly resulting in congestion for your game traffic, it might help you some. I doubt this is the case though