[SOLVED] emWeb- requested file gziped issue

JureLak · Jun 2nd 2023, 1:16pm

Hi,
Im testing webserver functions of emNet. I tried the addon for gzip compressed requests: IP_WEBS_ConfigFindGZipFiles.
I've noticed that this addon doesn't work for all file types. In my case that filetypes were .html.gz, .oga.gz and .mp3.gz.
Is there any way to add an option to make these files work ? Or am I doing something wrong ?

Best regards

SEGGER - Oliver · Jun 2nd 2023, 2:12pm

Hi Jure,

Can you please let me know which package (zip filename) you are using ?
I can then check if this is due to an old emWeb version used in this package or if there is something else that does not work as expected.

Can you please also let me know the exact IP_WEBS_ConfigFindGZipFiles() line that you are using and if there are different possible filesystems that can be used with this package (ReadOnly, emFile, Windows) please also mention what FS layer you are currently using.

Best regards,
Oliver Olligs
Product Manager

JureLak · Jun 3rd 2023, 9:58am

Hi Oliver,
I'm using latest ESPRO package 2.62 which uses emNet V3.50.2.

This is the exact line of code:

Source Code

IP_WEBS_ConfigFindGZipFiles(".gz",0);

Regarding FS I'm using ReadOnly mode. I used an example file IP_FS_ReadOnly.c from this package and edit it for my case. Gzip feature works fine for file types .js.gz, .png.gz, .svg.gz, .ttf.gz, .json.gz. ,...
But not for .html.gz, .oga.gz and .mp3.gz.

Best regards,
Jure

SEGGER - Oliver · Jun 5th 2023, 11:19am

Hi Jure,

For ".html.gz" this is caused by the configuration reported by _pfGetFileInfo() in the DynContent file.
As ".htm" and ".html" in our samples are meant to be parsed and filled in by our template system (the callbacks in the DynContent), by configuration no pre-gzipped file can be sent for them.

I just tested the feature with a "example.oga" file (actual OGG audio file) and it works as expected, delivering the pre-gzipped "example.oga.gz" if it is present.

Wireshark capture of the transfer when the pre-gzipped "example.oga.gz" does exist on the filesystem:

HTML Source Code

GET /example.oga HTTP/1.1
Host: 192.168.<HIDDEN>
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Pragma: no-cache
Cache-Control: no-cache
HTTP/1.1 200 OK
Content-Type: application/octet-stream
Server: emWeb
Expires: 1 JAN 2099 00:00:00 GMT
Last-Modified: 1 JAN 2009 00:00:00 GMT
Content-Encoding: gzip
Transfer-Encoding: chunked
5A0
......}d..example.oga...u\.A.0<.,.)]. ....(..tw.......HH....... !
....H
....}.....x..p]g..93'....=GG}...s....\?....B.!..>6........=.....F............."..._W8.?6.h..u...$......uoF...........f..I]....m...`.{.**.B...6_j..B_.(.ZE...z.a.m.E-..za-.Av.$.7..qf\..I.p= V..ET...K...qJ$.....B...q$.....rBS.p..V.% ".H...q.$h.:s..b...wR.~lbj:.XR..Q....#.....`b\ ....&"...^F....F9*<..7...ZR$.B.u...z.XR..Q.
<ETC.>

Display All

Wireshark capture of the transfer when the pre-gzipped "example.oga.gz" does not exist on the filesystem:

HTML Source Code

GET /example.oga HTTP/1.1
Host: 192.168.<HIDDEN>
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Pragma: no-cache
Cache-Control: no-cache
HTTP/1.1 200 OK
Content-Type: application/octet-stream
Server: emWeb
Expires: 1 JAN 2099 00:00:00 GMT
Last-Modified: 1 JAN 2009 00:00:00 GMT
Transfer-Encoding: chunked
5A0
OggS....................."...vorbis.....D........8........OggS..................c4...>..............vorbis....Lavf58.3.100........encoder=Lavc58.9.100 libvorbis..vorbis"BCV..@..$s.*F.s....BP...B.k..BL...2L[.%s.!..B.[(...U..@...Ax...A.!.%=X..'=.!..9x..iA.!..B.!..B.!.E9h..'A....08...8...E9X..'A. ..B....9.!.$5HP..9....,(...0....5(...0....B...I5...gAx..iA.!.$AH...A...FAX...9....A...*9... 4d.......(..(
<ETC.>

Display All

Here you can clearly read the OGG Vorbis header of the file.
The file as of ~26kBytes size, the first chunk transferred was always a full chunk, therefore in both cases 0x5A0 were transferred in the first chunk.
The actual size of the whole transfer differed between no gzip and pre-gzipped transfer.

Are you sure this does not work on your side for transferring the pre-gzipped file ?
Does it deliver the page if you do not have the uncompressed version at all ? This would be a clear indicator that the logic does find the pre-gzippped file.

When looking at the transfer via Wireshark, please make sure to right click => "Follow" => "TCP Stream".
With other methods Wireshark might show you decoded and otherwise processed content that is not actually what has been transferred packet wise. With Wireshark defaults you can not blindly trust what you see here.

Best regards,
Oliver

JureLak · Jun 5th 2023, 5:06pm

Hi Oliver,

For ".html.gz" this is caused by the configuration reported by _pfGetFileInfo() in the DynContent file.
As ".htm" and ".html" in our samples are meant to be parsed and filled in by our template system (the callbacks in the DynContent), by configuration no pre-gzipped file can be sent for them.

Thank you for pointing that out. I commented out that part from _pfGetFileInfo and now .html.gz and htm.gz are working.

But I still cant get .oga.gz and .mp3.gz to work. The uncompressed version is working normally.
I made a sample project for STM32H743ZI-Nucleo board so you can see it for yourself.
The project is based on segger.com/evaluate-our-softwa…cs/st-stm32h743zi-nucleo/ EVALUATION software.
I changed the webserver part. I did a simple html page with two audio files (audio controls).
Project is by default configured to use compressed files. You can simply change it to uncompressed by running a batch file SeggerEval_STM32H743_ST_STM32H743ZI_Nucleo_CortexM_SES_220601\IP\Setup\IP_FS\FS_RO_2018\m.bat.
You must also comment out #define GZIP in IP_FS_ReadOnly_2018.c file.

Here is zip file of project: https://we.tl/t-7PRwvgaico

Best regards,
Jure

JureLak · Jun 5th 2023, 6:12pm

Hi Oliver,
I just found out that compressed version of mp3 and oga are working if I request them individually and not from index.htm.
Actually, compressing already compressed files such as mp3 and oga makes no sense. So you can drop the research on that.
The important thing for me is that I can compress files like html, htm, css, js, svg,...
Thank you again for your help. I apologise for wasting your time due to my lack of knowledge and familiarity with your library.

Best regards,
Jure

SEGGER - Oliver · Jun 6th 2023, 10:00am

Hi Jure,

Nevermind. Others might benefit from this when looking for similar topics.

JureLak wrote:

I just found out that compressed version of mp3 and oga are working if I request them individually and not from index.htm.

Yes, that is actually how I tested this.

JureLak wrote:

Actually, compressing already compressed files such as mp3 and oga makes no sense. So you can drop the research on that.

Simplified, yes, it does not make sense and your browser knows that too and at least decided not to use it when the file is being embedded in an audio control in another page.

The more technical reason for this is most likely that the browser might open audio files as a possible stream source which is easier to handle if not compressed.
Easier for the browser and especially easier for the server as with a potential stream this could mean to compress an undefined length (live stream) on-the-fly.

Therefore the browser does not even try to request the audio file with gzip by not including it in an "Accept-Encoding:" field in its request or by omitting the field completely.

The first request for the page itself tells the server that it may use gzip encoding:

HTML Source Code

GET / HTTP/1.1
Host: 192.168.<HIDDEN>
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Pragma: no-cache
Cache-Control: no-cache
HTTP/1.1 200 OK
Content-Type: text/html
Server: emWeb
Expires: 1 JAN 2099 00:00:00 GMT
Last-Modified: 1 JAN 2009 00:00:00 GMT
Content-Encoding: gzip
Transfer-Encoding: chunked
<!DOCTYPE html>
<html>
<body>
<audio controls>
<source src="bell.oga" type="audio/ogg">
Your browser does not support the audio element.
</audio>
<audio controls>
<source src="bell2.mp3" type="audio/mpeg">
Your browser does not support the audio element.
</audio>
</body>
</html>

Display All

The embedded audio source requests the file without allowing gzip encoding, so it is actually the browser that does not want to use gzip at all in this case.

HTML Source Code

GET /bell.oga HTTP/1.1
Host: 192.168.<HIDDEN>
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0
Accept: audio/webm,audio/ogg,audio/wav,audio/*;q=0.9,application/ogg;q=0.7,video/*;q=0.6,*/*;q=0.5
Accept-Language: en-US,en;q=0.5
Range: bytes=0-
DNT: 1
Connection: keep-alive
Referer: http://192.168.11.84/
Accept-Encoding: identity
Pragma: no-cache
Cache-Control: no-cache

Display All

Best regards,
Oliver

[SOLVED] emWeb- requested file gziped issue

[SOLVED] emWeb- requested file gziped issue

Source Code

HTML Source Code

HTML Source Code

JureLak wrote:

JureLak wrote:

HTML Source Code

HTML Source Code

Share

Tags