[SOLVED] emWeb- requested file gziped issue

This site uses cookies. By continuing to browse this site, you are agreeing to our Cookie Policy.

  • [SOLVED] emWeb- requested file gziped issue

    Hi,
    Im testing webserver functions of emNet. I tried the addon for gzip compressed requests: IP_WEBS_ConfigFindGZipFiles.
    I've noticed that this addon doesn't work for all file types. In my case that filetypes were .html.gz, .oga.gz and .mp3.gz.
    Is there any way to add an option to make these files work ? Or am I doing something wrong ?

    Best regards
  • Hi Jure,

    Can you please let me know which package (zip filename) you are using ?
    I can then check if this is due to an old emWeb version used in this package or if there is something else that does not work as expected.

    Can you please also let me know the exact IP_WEBS_ConfigFindGZipFiles() line that you are using and if there are different possible filesystems that can be used with this package (ReadOnly, emFile, Windows) please also mention what FS layer you are currently using.

    Best regards,
    Oliver Olligs
    Product Manager
    Please read the forum rules before posting.

    Keep in mind, this is *not* a support forum.
    Our engineers will try to answer your questions between their projects if possible but this can be delayed by longer periods of time.
    Should you be entitled to support you can contact us via our support system: segger.com/ticket/

    Or you can contact us via e-mail.
  • Hi Oliver,
    I'm using latest ESPRO package 2.62 which uses emNet V3.50.2.

    This is the exact line of code:

    Source Code

    1. IP_WEBS_ConfigFindGZipFiles(".gz",0);

    Regarding FS I'm using ReadOnly mode. I used an example file IP_FS_ReadOnly.c from this package and edit it for my case. Gzip feature works fine for file types .js.gz, .png.gz, .svg.gz, .ttf.gz, .json.gz. ,...
    But not for .html.gz, .oga.gz and .mp3.gz.

    Best regards,
    Jure
  • Hi Jure,

    For ".html.gz" this is caused by the configuration reported by _pfGetFileInfo() in the DynContent file.
    As ".htm" and ".html" in our samples are meant to be parsed and filled in by our template system (the callbacks in the DynContent), by configuration no pre-gzipped file can be sent for them.

    I just tested the feature with a "example.oga" file (actual OGG audio file) and it works as expected, delivering the pre-gzipped "example.oga.gz" if it is present.

    Wireshark capture of the transfer when the pre-gzipped "example.oga.gz" does exist on the filesystem:

    HTML Source Code

    1. GET /example.oga HTTP/1.1
    2. Host: 192.168.<HIDDEN>
    3. User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0
    4. Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
    5. Accept-Language: en-US,en;q=0.5
    6. Accept-Encoding: gzip, deflate
    7. DNT: 1
    8. Connection: keep-alive
    9. Upgrade-Insecure-Requests: 1
    10. Pragma: no-cache
    11. Cache-Control: no-cache
    12. HTTP/1.1 200 OK
    13. Content-Type: application/octet-stream
    14. Server: emWeb
    15. Expires: 1 JAN 2099 00:00:00 GMT
    16. Last-Modified: 1 JAN 2009 00:00:00 GMT
    17. Content-Encoding: gzip
    18. Transfer-Encoding: chunked
    19. 5A0
    20. ......}d..example.oga...u\.A.0<.,.)]. ....(..tw.......HH....... !
    21. ....H
    22. ....}.....x..p]g..93'....=GG}...s....\?....B.!..>6........=.....F............."..._W8.?6.h..u...$......uoF...........f..I]....m...`.{.**.B...6_j..B_.(.ZE...z.a.m.E-..za-.Av.$.7..qf\..I.p= V..ET...K...qJ$.....B...q$.....rBS.p..V.% ".H...q.$h.:s..b...wR.~lbj:.XR..Q....#.....`b\ ....&"...^F....F9*<..7...ZR$.B.u...z.XR..Q.
    23. <ETC.>
    Display All


    Wireshark capture of the transfer when the pre-gzipped "example.oga.gz" does not exist on the filesystem:

    HTML Source Code

    1. GET /example.oga HTTP/1.1
    2. Host: 192.168.<HIDDEN>
    3. User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0
    4. Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
    5. Accept-Language: en-US,en;q=0.5
    6. Accept-Encoding: gzip, deflate
    7. DNT: 1
    8. Connection: keep-alive
    9. Upgrade-Insecure-Requests: 1
    10. Pragma: no-cache
    11. Cache-Control: no-cache
    12. HTTP/1.1 200 OK
    13. Content-Type: application/octet-stream
    14. Server: emWeb
    15. Expires: 1 JAN 2099 00:00:00 GMT
    16. Last-Modified: 1 JAN 2009 00:00:00 GMT
    17. Transfer-Encoding: chunked
    18. 5A0
    19. OggS....................."...vorbis.....D........8........OggS..................c4...>..............vorbis....Lavf58.3.100........encoder=Lavc58.9.100 libvorbis..vorbis"BCV..@..$s.*F.s....BP...B.k..BL...2L[.%s.!..B.[(...U..@...Ax...A.!.%=X..'=.!..9x..iA.!..B.!..B.!.E9h..'A....08...8...E9X..'A. ..B....9.!.$5HP..9....,(...0....5(...0....B...I5...gAx..iA.!.$AH...A...FAX...9....A...*9... 4d.......(..(
    20. <ETC.>
    Display All


    Here you can clearly read the OGG Vorbis header of the file.
    The file as of ~26kBytes size, the first chunk transferred was always a full chunk, therefore in both cases 0x5A0 were transferred in the first chunk.
    The actual size of the whole transfer differed between no gzip and pre-gzipped transfer.


    Are you sure this does not work on your side for transferring the pre-gzipped file ?
    Does it deliver the page if you do not have the uncompressed version at all ? This would be a clear indicator that the logic does find the pre-gzippped file.

    When looking at the transfer via Wireshark, please make sure to right click => "Follow" => "TCP Stream".
    With other methods Wireshark might show you decoded and otherwise processed content that is not actually what has been transferred packet wise. With Wireshark defaults you can not blindly trust what you see here.

    Best regards,
    Oliver
    Please read the forum rules before posting.

    Keep in mind, this is *not* a support forum.
    Our engineers will try to answer your questions between their projects if possible but this can be delayed by longer periods of time.
    Should you be entitled to support you can contact us via our support system: segger.com/ticket/

    Or you can contact us via e-mail.
  • Hi Oliver,
    For ".html.gz" this is caused by the configuration reported by _pfGetFileInfo() in the DynContent file.
    As ".htm" and ".html" in our samples are meant to be parsed and filled in by our template system (the callbacks in the DynContent), by configuration no pre-gzipped file can be sent for them.

    Thank you for pointing that out. I commented out that part from _pfGetFileInfo and now .html.gz and htm.gz are working.

    But I still cant get .oga.gz and .mp3.gz to work. The uncompressed version is working normally.
    I made a sample project for STM32H743ZI-Nucleo board so you can see it for yourself.
    The project is based on segger.com/evaluate-our-softwa…cs/st-stm32h743zi-nucleo/ EVALUATION software.
    I changed the webserver part. I did a simple html page with two audio files (audio controls).
    Project is by default configured to use compressed files. You can simply change it to uncompressed by running a batch file SeggerEval_STM32H743_ST_STM32H743ZI_Nucleo_CortexM_SES_220601\IP\Setup\IP_FS\FS_RO_2018\m.bat.
    You must also comment out #define GZIP in IP_FS_ReadOnly_2018.c file.

    Here is zip file of project: https://we.tl/t-7PRwvgaico

    Best regards,
    Jure
  • Hi Oliver,
    I just found out that compressed version of mp3 and oga are working if I request them individually and not from index.htm.
    Actually, compressing already compressed files such as mp3 and oga makes no sense. So you can drop the research on that.
    The important thing for me is that I can compress files like html, htm, css, js, svg,...
    Thank you again for your help. I apologise for wasting your time due to my lack of knowledge and familiarity with your library.

    Best regards,
    Jure
  • Hi Jure,

    Nevermind. Others might benefit from this when looking for similar topics.

    JureLak wrote:

    I just found out that compressed version of mp3 and oga are working if I request them individually and not from index.htm.
    Yes, that is actually how I tested this.

    JureLak wrote:

    Actually, compressing already compressed files such as mp3 and oga makes no sense. So you can drop the research on that.
    Simplified, yes, it does not make sense and your browser knows that too and at least decided not to use it when the file is being embedded in an audio control in another page.

    The more technical reason for this is most likely that the browser might open audio files as a possible stream source which is easier to handle if not compressed.
    Easier for the browser and especially easier for the server as with a potential stream this could mean to compress an undefined length (live stream) on-the-fly.

    Therefore the browser does not even try to request the audio file with gzip by not including it in an "Accept-Encoding:" field in its request or by omitting the field completely.

    The first request for the page itself tells the server that it may use gzip encoding:

    HTML Source Code

    1. GET / HTTP/1.1
    2. Host: 192.168.<HIDDEN>
    3. User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0
    4. Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
    5. Accept-Language: en-US,en;q=0.5
    6. Accept-Encoding: gzip, deflate
    7. DNT: 1
    8. Connection: keep-alive
    9. Upgrade-Insecure-Requests: 1
    10. Pragma: no-cache
    11. Cache-Control: no-cache
    12. HTTP/1.1 200 OK
    13. Content-Type: text/html
    14. Server: emWeb
    15. Expires: 1 JAN 2099 00:00:00 GMT
    16. Last-Modified: 1 JAN 2009 00:00:00 GMT
    17. Content-Encoding: gzip
    18. Transfer-Encoding: chunked
    19. <!DOCTYPE html>
    20. <html>
    21. <body>
    22. <audio controls>
    23. <source src="bell.oga" type="audio/ogg">
    24. Your browser does not support the audio element.
    25. </audio>
    26. <audio controls>
    27. <source src="bell2.mp3" type="audio/mpeg">
    28. Your browser does not support the audio element.
    29. </audio>
    30. </body>
    31. </html>
    Display All


    The embedded audio source requests the file without allowing gzip encoding, so it is actually the browser that does not want to use gzip at all in this case.

    HTML Source Code

    1. GET /bell.oga HTTP/1.1
    2. Host: 192.168.<HIDDEN>
    3. User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0
    4. Accept: audio/webm,audio/ogg,audio/wav,audio/*;q=0.9,application/ogg;q=0.7,video/*;q=0.6,*/*;q=0.5
    5. Accept-Language: en-US,en;q=0.5
    6. Range: bytes=0-
    7. DNT: 1
    8. Connection: keep-alive
    9. Referer: http://192.168.11.84/
    10. Accept-Encoding: identity
    11. Pragma: no-cache
    12. Cache-Control: no-cache
    Display All

    Best regards,
    Oliver
    Please read the forum rules before posting.

    Keep in mind, this is *not* a support forum.
    Our engineers will try to answer your questions between their projects if possible but this can be delayed by longer periods of time.
    Should you be entitled to support you can contact us via our support system: segger.com/ticket/

    Or you can contact us via e-mail.