[SOLVED] Large File Uploads with emWeb

This site uses cookies. By continuing to browse this site, you are agreeing to our Cookie Policy.

  • [SOLVED] Large File Uploads with emWeb

    Hello,

    I have an order in process for Embedded Studio Pro so I don't have paid support quite yet but will soon. In the meantime, I have a question concerning file uploads with the emWeb Server. It seems that the WebS InBuffer might need to be as large as the file that I am uploading? When I upload about a 3KByte file I get the log below that the InBuffer is too small for POST data. When I upload a 10 byte text file it appears to work better with no InBuffer warning but the file is not actually saved to the file system when checked running the FTP Server separately (probably a different issue?). What would be the proper way to upload files up to a maximum of 1 Megabyte using emWeb? The goal would be for customers to upload new firmware files. I thought maybe using the IP_RxTask might help but I still get the error. Best Regards.


    4:807 MainTask - INIT: embOS/IP init started. Version 3.31a
    4:807 MainTask - *********************************************************************
    4:807 MainTask - * embOS/IP Configuration *
    4:807 MainTask - *********************************************************************
    4:807 MainTask - * IP_DEBUG: 2
    4:807 MainTask - * Memory added: 24576 bytes
    4:808 MainTask - * Buffer configuration:
    4:808 MainTask - * 12 buffers of 256 bytes
    4:808 MainTask - * 6 buffers of 1516 bytes
    4:808 MainTask - * TCP Tx/Rx window size per socket: 4380/4380 bytes
    4:808 MainTask - * Number of interfaces added: 1
    4:808 MainTask - * Interface #0 configuration:
    4:808 MainTask - * Type: ETH
    4:808 MainTask - * MTU: 1500
    4:808 MainTask - * HW addr.: 00:22:C7:FF:FF:FF
    4:809 MainTask - * IPv4 static addr.: 10.135.246.111
    4:809 MainTask - *********************************************************************
    4:829 MainTask - INIT: Link is down
    4:829 MainTask - DRIVER: Found PHY with Id 0x2000 at addr 0x0
    4:830 MainTask - INIT: Init completed
    4:830 IP_Task - INIT: IP_Task started
    4:830 IP_RxTask - INIT: IP_RxTask started
    6:829 IP_Task - LINK: Link state changed: Full duplex, 100MHz
    6:830 PTPClient - Correction applied onto state "uncalibrated": 1551196467s, 917091479ns
    6:830 PTPClient - Not synced
    6:831 IP_WebServer - WEBS: Using a memory pool of 10996 bytes for 2 connections.
    7:207 IP_Task - NDP: Link-local IPv6 addr.: FE80:0000:0000:0000:0222:C7FF:FEFF:FFFF added to IFace: 0
    11:830 PTPClient - Not synced
    16:830 PTPClient - Not synced
    21:830 PTPClient - Not synced
    26:830 PTPClient - Not synced
    28:900 Webserver Child - WebS: Get /Upload.cgi
    28:901 Webserver Child - *** Warning *** WebS: InBuffer too small for POST data.

    //
    // Webserver buffer sizes from WEBS_Conf.h
    //
    #define WEBS_IN_BUFFER_SIZE 2920 // Buffer size should be at least 256 bytes.
    #define WEBS_OUT_BUFFER_SIZE 1460 // Buffer size can be decreased. To small buffers will result in a
    // lack of performance and decreases the quality of service.
    // Default is max. MTU(1500 bytes) - best case IPv4/TCP headers (40 bytes).
    #define WEBS_PARA_BUFFER_SIZE 512 // Required for dynamic content parameter handling.
    #define WEBS_FILENAME_BUFFER_SIZE 64


    //
    // Maximum allowed root path length of the Web server
    // in multiples of a CPU native unit (typically int).
    // If the root path of the Web server is the root of
    // your media you can comment out this define or set
    // it to zero.
    // Example:
    // For the root path "/httpdocs" the define needs
    // to be set to at least 9 . As this is not a multiple
    // of an int, set it to 12.
    //
    #define WEBS_MAX_ROOT_PATH_LEN 24 /*12*/

    //Defines from TASK Start Code
    //
    // Web server and IP stack
    //
    #define MAX_CONNECTIONS 2
    #define BACK_LOG 20
    #define MAX_CONNECTION_INFO 10
    #define IDLE_TIMEOUT 1000 // Timeout [ms] after which the connection will be closed if no new data is received.
    #define SERVER_PORT 80
    #define CHILD_ALLOC_SIZE 5498 // NumBytes required from memory pool for one connection. Should be fine tuned according
    // to your configuration using IP_WEBS_CountRequiredMem() .


    #ifndef STACK_SIZE_SERVER
    #define STACK_SIZE_SERVER (2200 + APP_TASK_STACK_OVERHEAD)
    #endif


    #ifndef TASK_STACK_SIZE_WEBSERVER_TASK
    #define TASK_STACK_SIZE_WEBSERVER_TASK (2048 + APP_TASK_STACK_OVERHEAD)
    #endif
  • Hello Jim,

    Uploads are processed in chunks of the InBuffer until the complete file has been uploaded.
    Therefore you do not need to have an InBuffer size equal or larger than your file that you want to upload.

    The reason why you end up with this message is most likely that you did not enable upload in the webserver sample.
    In this case the POST content is tried to be processed with the form handling code which is easily exhausted by an upload content.

    Please check for "IP_WEBS_AddUpload()" in the sample. It is commented out by default as not everyone needs uploading
    and it adds quite some code for the handling that is not necessary if you do not want to upload, which is why it is not enabled by default.


    Are you able to upload your bigger file(s) successfully when you enable upload using "IP_WEBS_AddUpload()" ?

    Best regards,
    Oliver Olligs
    emNet Product Manager
    Please read the forum rules before posting.

    Keep in mind, this is *not* a support forum.
    Our engineers will try to answer your questions between their projects if possible but this can be delayed by longer periods of time.
    Should you be entitled to support you can contact us via our support system: segger.com/ticket/

    Or you can contact us via e-mail.
  • Hi Oliver,

    Yes "IP_WEBS_AddUpload()" is uncommented. I set a breakpoint and the function appears to execute.

    Regards,

    Jim

    //
    // Assign file system
    //
    //_pFS_API = &IP_FS_ReadOnly; // To use a a real filesystem like emFile replace this line.
    _pFS_API = &IP_FS_emFile; // Use emFile
    IP_WEBS_AddUpload(); // Enable upload
    //
    // Configure buffer size.
    //
    IP_MEMSET(&BufferSizes, 0, sizeof(BufferSizes));
    BufferSizes.NumBytesInBuf = WEBS_IN_BUFFER_SIZE;
    BufferSizes.NumBytesOutBuf = IP_TCP_GetMTU(_IFaceId) - 72; // Use max. MTU configured for the last interface added minus worst case IPv4/TCP/VLAN headers.
    // Calculation for the memory pool is done under assumption of the best case headers with - 40 bytes.
    BufferSizes.NumBytesParaBuf = WEBS_PARA_BUFFER_SIZE;
    BufferSizes.NumBytesFilenameBuf = WEBS_FILENAME_BUFFER_SIZE;
    BufferSizes.MaxRootPathLen = WEBS_MAX_ROOT_PATH_LEN;
    //
    // Configure the size of the buffers used by the Webserver child tasks.
    //
    IP_WEBS_ConfigBufSizes(&BufferSizes);
  • Some additional info. I'm running the web server on the Cortex M4 core of an STM32H745 duo core processor. I don't think that should make a difference but I'd thought I'd mention it.

    With uploading a 10-byte text file using Upl.htm (non-Ajax) I get this response in the web browser: Upload of "----------------------------4426402082890354833656916082%0D%0AContent-Disposition%3A%20form-data%3B%20name%3D%22Data%22%3B%20filename%3D%22Upload.txt%22%0D%0AContent-Type%3A%20text%2Fplain%0D%0A%0D%0A123456789A%0D%0A-----------------------------4426402082890354833656916082--%0D%0A" not successful!

    With uploading a 3 kilobyte file and using the Upl_AJAX.htm page a message box pops up in the browser that shows the device ip address and "OnError". I also get the InBuffer too small for POST data warning in the Embedded Studio debug terminal.

    261:972 Webserver Child - WebS: Get /Upl_AJAX.htm
    266:754 PTPClient - Not synced
    271:299 Webserver Child - WebS: Get /Upload.cgi
    271:300 Webserver Child - *** Warning *** WebS: InBuffer too small for POST data.
  • Hi Jim,

    Can you please let me know from what components the project was assembled ?

    CodeRunner wrote:

    4:807 MainTask - INIT: embOS/IP init started. Version 3.31a
    This is an old beta version (odd minor numbers in the version are beta/pre-release) and I can not actually find why you have it.
    Do you have a different license where this has been taken from ?

    CodeRunner wrote:

    6:830 PTPClient - Correction applied onto state "uncalibrated": 1551196467s, 917091479ns
    Also you have recently licensed the PTP client, however according to my information you got it for for v3.40e and not the used v3.31a .
    Using this with different versions might conflict with some structures and even cause memory corruption if wrong structure offsets might be used.
    Not sure if this is really what is happening but it is a bad idea to mix different versions of modules like PTP that are not designed to be used stand-alone.

    The webserver for example is stand-alone and is abstracted from the stack via the socket API.
    PTP might still overwrite some memory which would not be good...

    Best regards,
    Oliver
    Please read the forum rules before posting.

    Keep in mind, this is *not* a support forum.
    Our engineers will try to answer your questions between their projects if possible but this can be delayed by longer periods of time.
    Should you be entitled to support you can contact us via our support system: segger.com/ticket/

    Or you can contact us via e-mail.
  • Thanks. I just received my "licensed" PTP source and Embedded Studio Pro activation yesterday. I had been working from an evaluation PTP demo for the STM32H743 eval board that I received a long time ago. I will work on updating everything to the most recent libraries and source and will reply with the results.
  • I think I am getting closer. I figured out how to enable individual Embedded Studio Pro includes and libraries in the project options. Now when I build the app the linker can't seem to link to the libraries. Is there a linker option or possible path issue with my project that is causing the linker not to find symbols from Embedded Studio Pro?

    1> Linking Start_STM32H745_Nucleo_M4.elf
    1>
    1> Diagnostics:
    1> error: undefined symbol: IP_GetPTPClock
    1> error: undefined symbol: IP_Log
    1> error: undefined symbol: IP_Panic
    1> error: undefined symbol: IP_WEBS_X_SampleConfig
    1> error: undefined symbol: IP_Warn
    1> error: undefined symbol: OS_AssertCPUInISRMode
    1> error: undefined symbol: OS_COM_GetNextChar
    1> error: undefined symbol: OS_COM_OnRx
    1> error: undefined symbol: OS_COM_OnTx
    1> error: undefined symbol: OS_EVENT_Create
    1> error: undefined symbol: OS_EVENT_Delete
    1> error: undefined symbol: OS_EVENT_GetBlocked
    1> error: undefined symbol: OS_EVENT_GetTimed
    1> error: undefined symbol: OS_EVENT_Set
    1> error: undefined symbol: OS_EnterRegionFunc
    1> error: undefined symbol: OS_Global
    1> error: undefined symbol: OS_INFO_GetVersion
    1> error: undefined symbol: OS_InitCalled
    1> error: undefined symbol: OS_Init_VFP
    1> error: undefined symbol: OS_MUTEX_Create
    1> error: undefined symbol: OS_MUTEX_GetValue
    1> error: undefined symbol: OS_MUTEX_LockBlocked
    1> error: undefined symbol: OS_MUTEX_Unlock
    1> error: undefined symbol: OS_Running
    1> error: undefined symbol: OS_SEMAPHORE_GiveMax
    1> error: undefined symbol: OS_Status
    1> error: undefined symbol: OS_SwitchFromInt
    1> error: undefined symbol: OS_TASKEVENT_GetTimed
    1> error: undefined symbol: OS_TASKEVENT_Set
    1> error: undefined symbol: OS_TASK_AddTerminateHook
    1> error: undefined symbol: OS_TASK_CreateEx_D
    1> error: undefined symbol: OS_TASK_Create_D
    1> error: undefined symbol: OS_TASK_Delay
    1> error: undefined symbol: OS_TASK_IsTask
    1> error: undefined symbol: OS_TASK_LeaveRegion
    1> error: undefined symbol: OS_TASK_SetName
    1> error: undefined symbol: OS_TASK_SetPriority
    1> error: undefined symbol: OS_TASK_Terminate
    1> error: undefined symbol: OS_TICK_Handle
    1> error: undefined symbol: OS_TIME_ConfigSysTimer
    1> error: undefined symbol: OS__Start
    1> error: undefined symbol: WebsSample_Application
  • Hello,

    From the error messages it appears that the ESPRO Library options are not set correctly corresponding to your source setup.
    Now that you are in valid support period could you open a support ticket in the Embedded Studio support group as explained in my signature?

    Could you provide additional details about how you set up/created your project exactly?
    What eval board are you using as a base?

    If I see correctly you told our sales that you are using the H743 Nucleo Board, not the Eval.
    So which one is it? Because we are working on a BSP for your specific platform, but would need to get information about which board you are using.

    Then with the finished ESPRO BSP it should be much easier to get started with your project.

    This thread will be closed now to avoid information being lost between channels.

    Best regards,
    Nino
    Please read the forum rules before posting.

    Keep in mind, this is *not* a support forum.
    Our engineers will try to answer your questions between their projects if possible but this can be delayed by longer periods of time.
    Should you be entitled to support you can contact us via our support system: segger.com/ticket/

    Or you can contact us via e-mail.