Development Journal of a Raytracer - Web

Compression of HTTP Responses - 2

nospam@example.com (Josef Meixner) — Sun, 14 Mar 2010 21:54:53 +0100

I also worked on the HTTPD. Compression is still not implemented, but today I added the code to create the necessary "content-encoding" header and also wrote the support code to send the compressed data. All that is missing now is the function which compresses the data.

I looked into the zlib documentation, but I will need a bit of time to get that correctly implemented. One of the bigger issues is the amount of memory needed for compression. By default the zlib allocates about 256K Bytes per compression channel. I think that is too high for what I try to do, most web pages are by far smaller than that and so I hope that the compression actually doesn't suffer when I reduce the amount allocated. I think, that using 1GB per 4000 HTTP responses is quite high and I want to be able to have such an amount in flight. As my server is going to be single threaded it can actually happen, that connections waiting for operations e.g. on a database to complete can already have a compression channel and so it might be possible to queue up a lot of channels and so take away a lot of memory.

Compression of HTTP Responses

nospam@example.com (Josef Meixner) — Sat, 06 Mar 2010 23:20:21 +0100

Today I also worked on the HTTPD. I have extended the response- and request-classes so that the request can tell the response, that the client (i.e. the browser requesting a page) can handle compressed pages. As the evaluation, if a client understands compression is costly, I added a virtual function to the response so an implementation can inform the request, that it can't compress the contents and so it doesn't have to look into the header. Currently the error response doesn't return any data, so it obviously doesn't need to support compression. The same would be true of binary data like images, but I haven't defined a response to handle that yet, currently I can only send texts.

Restarted work on the HTTPD

nospam@example.com (Josef Meixner) — Sun, 28 Feb 2010 22:13:41 +0100

It has been forever that I really worked on the HTTPD, I have sometimes done some small stuff, but mostly to get it compiling again. Last weekend I managed to update my SuSE 11.1 installation to 11.2 (astoundingly the updater really works) and it finally seems to support my system well enough that I can work. The network is detected with the right speed (in 11.1 the network came up with 10Mb/s) and even my audio card works again.

I am still at the point to add compression to the HTTPD, but I first added something else. As I want to write web services in C++ I need some support to make it easier to create pages. The first addition was, to have a class which creates the HTML-header (containing the DTD, the title and so on). I will extend it later so I can also pass the location of CSS- or JS-files, that way I can write it down correctly once and don't need to remember or look up how to do it. Yes, I know I am lazy.

With that I extended my first test page to contain a bit more text. Next weekend I will try to finally add primitive support for compression.

Getting the HTTPD working again

nospam@example.com (Josef Meixner) — Wed, 20 May 2009 22:57:23 +0200

I haven't worked on the HTTPD forever. Today I finally got it to compile again on a virtual machine. No big changes, just some missing includes needed to be added. I tried to find out, if it would be possible to develop the program in a virtual machine and it seems to work quite well. I exported the directory containing the sources with samba and could then edit the code in Visual Studio (I don't really like to use editors in VMs, they always lag a bit).

So I hope I will now finally make the changes I intended a long time ago. The thing I want to do as next thing is to add compression to the http-connection.

New Entry Page

nospam@example.com (Josef Meixner) — Sun, 30 Dec 2007 19:27:49 +0100

Today I finally uploaded a new entry page to my domain. I worked quite long on the scene and you might understand why I did some of the changes in the last two months (yes, it took that long). There are still some problems with the image, but for now it has to be sufficient. I would much rather have a longer menu to display.

I still don't like to create HTML pages, I think it is a pain. Especially getting it to work in Internet Explorer turned out to be a problem again. I don't even know if it currently works, after I had checked the page in IE 6, fixed the problems (including the whitespace bug) and then checked it in IE7, I found some problems with the way it displayed in Firefox. So I removed some CSS (and hope it didn't destroy rendering in IE) and had to add a conditional comment for two statements for IE. Those I have not yet tested in IE, but I hope it still works.

New Template

nospam@example.com (Josef Meixner) — Thu, 17 May 2007 15:02:58 +0200

I just uploaded the changed template. I have removed my small logo, although I really liked it, but it didn't fit in any way to the image. The image is a bit big, I might change the jpg-compression a bit. I am unsure if I should also change the rest, but I think it might make sense to give it a bit different background and I might change the headers of the plugins and the posts. But I am not sure about that, yet. I will see.

What I have found very useful in doing this was firebug, a plugin for firefox. I totally love the ability to change CSS on the currently displayed page. To adjust font sizes, margins, frames and colors there is nothing better. Also to just try, if the text looks better to the right (text-align: right;) it is great as you don't have to find it in the CSS, change it, reload the page. But you just hit the inspect button, click the element and add a new property. Highly recommended.

Only thing it doesn't have is the ability to save the CSS. You can change into the CSS tab and select all and save it. I don't know if that works without flaws, though, I haven't tried it. I don't know what happens if you actually use multiple style sheets. I only made very few changes, mostly had to find a way to align the title of the blog.

Preparations for New Style

nospam@example.com (Josef Meixner) — Tue, 08 May 2007 22:21:17 +0200

I today actually did the first step to prepare this blog for a new design. No I still like this one, I only want to change the header. As I wrote in the todo-list I want to use one of my rendered images at the place of the blue bar. I will also make it a big higher so I can actually fit something into it.

For that to happen I will have to modify the template a bit and also have to find out, if the title text stays readable and how to keep my logo. I know, it isn't much, but I actually like it.

For that I copied the current database and the application to my intranet server and changed some settings so it runs on it. So I can test it before I put it here. It will probably take some days (I hope I at least manage to do that in May).

Security - Addendum to 'Recover Password'

nospam@example.com (Josef Meixner) — Fri, 12 Jan 2007 22:54:35 +0100

I recently wrote a bit about the 'Recover Password' stuff you find on many websites and said, that I would target that thing to break into, as I would guess that there I would get much further with a dictionary attack. Yesterday Bruce Schneier posted an essay he had written for Wired about how to choose a secure password. There is a sentence which I found in line with my considerations:

Your encryption program's key-escrow system is almost certainly more vulnerable than your password, as is any "secret question" you've set up in case you forget your password.

Seems as if I am not alone in thinking those things are dangerous.

Security - 4 - Security Placebo 2

nospam@example.com (Josef Meixner) — Sun, 07 Jan 2007 11:55:42 +0100

I didn't intend to write about security placebos again so soon, but I found such a glaring example I have to share. Steffan Esser from the Hardened PHP Project has a blog about security problems with PHP called PHP Security Blog. He recently wrote about a new PHP plugin called ext/filter and suggested to not use it.

Continue reading "Security - 4 - Security Placebo 2"

Compression Design

nospam@example.com (Josef Meixner) — Sat, 06 Jan 2007 09:38:20 +0100

As you might know, I still haven't added compression to my HTTP-layer. The reasons are not only that I hadn't implemented parsing to find out, if the client can even understand it, but also because it is a very important part in my eyes and I want to do it right. I want my layer to be very fast and I have hopes I can reach that. One of the traditional speed killers in networking is excessive copying of data. So if I can save on copies, it will have a positive effect, as long as it doesn't end in making the rest more complicated.

Continue reading "Compression Design"

Request-Parsing Finished (and a Rant about HTTP)

nospam@example.com (Josef Meixner) — Fri, 05 Jan 2007 22:26:01 +0100

I worked a lot on the HTTP implementation today. Now the parsing of requests should be finished. The compression still isn't implemented, because I fist had to implement the parsing of the Accept-Encoding header. This header is used by the client to inform a server if and which compression methods it supports.

Continue reading "Request-Parsing Finished (and a Rant about HTTP)"

Connection control working

nospam@example.com (Josef Meixner) — Sun, 31 Dec 2006 21:09:11 +0100

I Implemented the connection header. Also fixed some issues with timeouts and sometimes the closing of a connection wasn't detected properly. And I found two big bugs in the receive buffer when testing closing when the maximum number of requests was reached.

Next will be support for compression. I already looked a bit into the zlib documentation. I used it before in my raytracer (I wrote a streambuf-implementation to store my data files as compressed XML). There is something I don't know yet, it seems as if it is possible to compress in multiple stages. This might be an interesting feature some day, as it would make it possible to create a web page where big parts are static and could be precompressed and only the variable parts would need to be compressed for each request. I don't know if that is possible, though.

Timeouts are working

nospam@example.com (Josef Meixner) — Sat, 30 Dec 2006 22:27:39 +0100

Finally found some time to implement the timeouts in the HTTPD. Now I am only missing correct signaling of the connection handling stuff (basically the Connection: close when the maximum number of requests that will be served on one connection has been reached) and I would like to implement compression. That means adding the code to compress responses and the correct evaluation of the flags to know if we can send compressed data.

Then I will try to set up the benchmark I wanted to do for some time. I still have to compile a lighttpd with php support. I at least haven't seen any way to only install php without also installing Apache. As I am not really interested in Apaches behaviour I don't want to install it.

Security - 3 - Security Placebo

nospam@example.com (Josef Meixner) — Fri, 29 Dec 2006 21:32:00 +0100

Last time I wrote about the things I learnt recently about security. That the security industry has failed, that the common security stuff doesn't work and that there is no sign it will soon. I linked to a nice post about the biggest mistakes you can make when implementing security. I absolutely agree with that and I even have an explanation why it fails, nobody wants to spend thought and time to implement it properly. It has been written again and again and again, security is no product but a process. Perhaps it is easier to understand for me as I worked at the university in a group working in reliability research. There the same principles apply, you can't make a system reliable as an afterthought. It won't work. No, never.

Continue reading "Security - 3 - Security Placebo"

Security - 2 - A Complete Failure

nospam@example.com (Josef Meixner) — Sun, 17 Dec 2006 20:41:00 +0100

Last time I wrote about the many holes of a web site, there were two more incidents, they didn't fully close the hole to invite yourself to another group. They tried to fix it with a checksum in a hidden field. Obviously they still don't check it on the server side (don't ask me why, it is the only thing you can do to make sure it wasn't tempered with) so I guess in the checksum is the user id of the one issuing the invitation as you needed to be a group moderator (which is easy to become, everybody can start a new group and thereby becoming a moderator). But I guess the group to invite to wasn't included in the checksum. The second was a SQL injection hole on the "forgot password" functionality.

But now for something different. Writing about security holes isn't so much of my interest than finding out, how to not allow them to happen. That was one of the things I did in the week I wasn't very active. There were some articles I read and I have to say, they expressed something I somehow knew but couldn't put into words. It seems, the security industry has managed to somehow get off track. Since it became big business it seems that the interest is more in selling something which we know won't work than in fixing problems. For me the point in time when that became clear was, when Microsoft Corp. started to distribute their malware removal tool. Sorry, but I expect the one responsible for the holes to fix them and not sell me (or for now gift me with) some duct tape.

Continue reading "Security - 2 - A Complete Failure"