We use ultra-deep ultraviolet VLT/VIMOS intermediate-band and VLT/FORS1 narrow-band imaging in the GOODS Southern field to derive limits on the distribution of the escape fraction (f_esc) of ionizing radiation for L >~ L*(z=3) Lyman Break Galaxies (LBGs) at redshift 3.4--4.5. Only one LBG, at redshift z=3.795, is detected in its Lyman continuum (LyC; S/N~5.5), the highest redshift galaxy currently known with a direct detection. Its ultraviolet morphology is quite compact (R_eff=0.8, kpc physical). Three out of seven AGN are also detected in their LyC, including one at redshift z=3.951 and z850 = 26.1. From stacked data (LBGs) we set an upper limit to the average f_esc in the range 5%--20%, depending on the how the data are selected (e.g., by magnitude and/or redshift). We undertake extensive Monte Carlo simulations that take into account intergalactic attenuation, stellar population synthesis models, dust extinction and photometric noise in order to explore the moments of the distribution of the escaping radiation. Various distributions (exponential, log-normal and Gaussian) are explored. We find that the median f_esc is lower than ~6% with an 84% percentile limit not larger than 20%. If this result remains valid for fainter LBGs down to current observational limits, then the LBG population might be not sufficient to account for the entire photoionization budget at the redshifts considered here, with the exact details dependent upon the assumed ionizing background and QSO contribution thereto. It is possible that f_esc depends on the UV luminosity of the galaxies, with fainter galaxies having higher f_esc, and estimates of f_esc from a sample of faint LBG from the HUDF (i775<28.5) are in broad quantitative agreement with such a scenario.