System.FormatException: Invalid character in a Base-64 string...

Problem:
Enter the URL to a small image file (or any file), retrieve the file and encode the data as base64, then decode the base64 to get the original files data.

When the page below is executed, the file is retrieved and encoded, however when the decode_base64() subroutine is called, the system displays the following exception:

System.FormatException: Invalid character in a Base-64 string.
at System.Convert.FromBase64String(String s)
at ASP.test_base64_aspx.decode_base64()

Obviously, there is an invalid character in the Base-64 string -- the strange thing is if I increase "k_buf_size" from 4096 to something like 32000 (in the encode_base64 subroutine) and execute the page everything works just fine!!

So my question is, why will this code NOT work with smaller buffer sizes?? Hopefully, I have made a simple programming error that someone will point out as I have stared at this for too long ...

ANY help would be appreciated!

<%@dotnet.itags.org. Page Language="VB" Explicit="true" Strict="true" Trace="true" %>

<%@dotnet.itags.org. Import Namespace="System.IO" %>
<%@dotnet.itags.org. Import Namespace="System.Data" %>
<%@dotnet.itags.org. Import Namespace="System.Net" %>

<script language="VB" runat="server">

' The base64 encoded data ...
private m_encoded_data as StringBuilder

' The size of the file that was encoded ...
private m_file_size as Integer

'----------------
'
private sub do_test( Sender as Object, e as EventArgs )

' Init ...
m_encoded_data = new StringBuilder()
m_file_size = 0

' Attempt to encode the input file ...
encode_base64()

' If a file was encoded, attempt to decode ...
if ( m_encoded_data.length > 0 ) then

decode_base64()

end if

end sub

'----------------
'
private sub encode_base64()

const k_buf_size as Integer = 4096

dim bytes_read as Integer
dim n as Integer

dim err_msg as String
dim s as String
dim url as String

dim sr as Stream

dim wc as WebClient

Trace.Warn( ">", "==============" )
Trace.Warn( ">", "encode_base64:" )

' Get URL ...
url = fld_url.text.Trim()
if ( url.length = 0 )
exit sub
end if

' Attempt to get the data file and encode as base 64 ...
try

wc = Nothing
sr = Nothing
m_file_size = 0

' Create space to hold data ...
dim data_array( k_buf_size ) as Byte

' Create web client ...
wc = new WebClient()

' Create stream from URL ...
sr = wc.OpenRead( url )

' Loop to encode data stream ...

' Read a chunk from the data steam into array ...
bytes_read = sr.Read( data_array, 0, k_buf_size )

do while ( bytes_read > 0 )

Trace.Warn( ">", "bytes_read = " & bytes_read.ToString() )

m_file_size = m_file_size + bytes_read

' Convert data chunk to base 64 string. Remove any padding characters (=)
' that may have been added ...
s = System.Convert.ToBase64String( data_array, 0, bytes_read )
n = s.IndexOf( "=" )

if ( n > 0 ) then
m_encoded_data.Append( s.Substring( 0, n ) )
else
m_encoded_data.Append( s )
end if

bytes_read = sr.Read( data_array, 0, k_buf_size )

loop

' Encoded output length must be a multiple of 4 bytes, so pad if necessary ...
n = m_encoded_data.length Mod 4
if ( n <> 0 ) then
m_encoded_data.Append( new String( "="C, 4 - n ) )
end if

' Clean up ...
data_array = Nothing

Trace.Warn( ">", "File size = " & m_file_size.ToString() )
Trace.Warn( ">", "Encoded data length = " & m_encoded_data.length.ToString() )

catch ex as Exception
Trace.Warn( ">", ex.ToString() )

finally
if ( not sr is Nothing ) then
sr.Close()
sr = Nothing
end if
if ( not wc is Nothing ) then
wc.Dispose()
wc = Nothing
end if

end try

end sub

'----------------
'
private sub decode_base64()

dim data_array() as Byte

Trace.Warn( ">", "==============" )
Trace.Warn( ">", "decode_base64:" )

' Attempt to decode from base64 ...
try

data_array = Convert.FromBase64String( m_encoded_data.ToString() )

Trace.Warn( ">", "data_array.length = " & data_array.length.ToString() )

if ( data_array.length = m_file_size ) then
Trace.Warn( ">", "DECODE OK" )
else
Trace.Warn( ">", "DECODE FAILED!" )
end if

catch ex as Exception
Trace.Warn( ">", ex.ToString() )

end try

end sub
</script>
<html>
<head>
<title>Attempt to encode/decode using Base 64</title>
</head>
<body>
<blockquote>

<form runat="server">
Enter the URL to a file, preferably a small image JPG or GIF file.
<hr noshade size="1">
<table width="90%" border="0" cellpadding="2" cellspacing="1">
<tr>
<td width="20%" align="right" valign="top" >
URL:
</td>
<td width="80%" align="left" valign="top" >
<asp:TextBox id="fld_url" text="" size="100" maxlength="200" runat="server" />
</td>
</tr>
</table>
<hr noshade size="1">
<table width="90%" border="0" cellpadding="2" cellspacing="1">
<tr>
<td width="20%"> </td>
<td width="80%" valign="top">
<asp:Button OnClick="do_test" text="Do Base64 Test" runat="server" />
</td>
</tr>
</table>
</form>

</blockquote>
</body>
</html>

Ok - here is one solution (may not be the best, but it works)...

The problem with base 64 encoding is that one has to be careful when reading data in "chunks" and then encoding each chunk (as base 64) and then appending each chunk to form the final encoded string -- this is what I was doing initially. The code would work whenever I changed the buffer size to anything larger than the file being read, ie: whenever the entire file was read in one chunk. In any case, the solution is shown below for anyone that comes across this problem. Please note that the solution below limits the file sizes that can be encoded to 2 megabytes.

<%@. Page Language="VB" Explicit="true" Strict="true" Trace="true" %>

<%@. Import Namespace="System.IO" %>
<%@. Import Namespace="System.Data" %>
<%@. Import Namespace="System.Net" %>

<%@. Import Namespace="rentsys" %>

<script language="VB" runat="server">

' The base64 encoded data ...
private m_encoded_data as String

' The size of the file that was encoded ...
private m_file_size as Long

'----------------
'
private sub do_test( Sender as Object, e as EventArgs )

' Init ...
m_encoded_data = String.Empty
m_file_size = 0

' Attempt to encode the input file ...
encode_base64()

' If a file was encoded, attempt to decode ...
if ( m_encoded_data.length > 0 ) then

decode_base64()

end if

end sub

'----------------
'
private sub encode_base64()

' Maximum file size is 2 MB ...
const k_max_file_size as Long = 2000000

' Buffer used to read file in chunks ...
const k_data_buf_size as Integer = 4096
dim data_buf( k_data_buf_size ) as Byte

dim bytes_read as Integer

dim err_msg as String
dim s as String
dim url as String

dim ms as MemoryStream

dim sr as Stream

dim wc as WebClient

Trace.Warn( ">", "==============" )
Trace.Warn( ">", "encode_base64:" )

' Get URL ...
url = fld_url.text.Trim()
if ( url.length = 0 )
exit sub
end if

' Get the size of the remote file, unfortunately, the only way
' to reliably get this is to read the entire file. We could issue
' an HTTP HEAD request to get the content length, but some servers
' may not return file headers ...
try

wc = Nothing
sr = Nothing
m_file_size = 0

' Create web client ...
wc = new WebClient()

' Create stream from URL ...
sr = wc.OpenRead( url )

' Loop to determine file size ...
bytes_read = sr.Read( data_buf, 0, k_data_buf_size )
do while ( bytes_read > 0 )
m_file_size = m_file_size + bytes_read
bytes_read = sr.Read( data_buf, 0, k_data_buf_size )
loop

catch ex as Exception
Trace.Warn( ">", ex.ToString() )

finally
if ( not sr is Nothing ) then
sr.Close()
sr = Nothing
end if
if ( not wc is Nothing ) then
wc.Dispose()
wc = Nothing
end if

end try

' Check if we can handle the file size ...
Trace.Warn( ">", "File size = " & m_file_size.ToString() )

if ( m_file_size > k_max_file_size ) then
Trace.Warn( ">", "File size exceeds maximum allowed" )
exit sub
end if

' Attempt to read the file contents and encode as base 64 ...
try

wc = Nothing
sr = Nothing
ms = Nothing

' Create web client ...
wc = new WebClient()

' Create stream from URL ...
sr = wc.OpenRead( url )

' Create memory stream to hold file contents ...
ms = new MemoryStream()

' Loop to read data into memory stream. Note that we
' CANNOT convert each chunk read into base 64 and append
' to a string as this would lead to an invalid base 64
' string due to the nature of the algorithm and padding
' characters that would be present in the final string ...

bytes_read = sr.Read( data_buf, 0, k_data_buf_size )
do while ( bytes_read > 0 )

ms.Write( data_buf, 0, bytes_read )

bytes_read = sr.Read( data_buf, 0, k_data_buf_size )

loop

' Convert the data to base 64 ...
m_encoded_data = Convert.ToBase64String( ms.ToArray() )

catch ex as Exception
Trace.Warn( ">", ex.ToString() )

finally
if ( not ms is Nothing ) then
ms.Close()
ms = Nothing
end if
if ( not sr is Nothing ) then
sr.Close()
sr = Nothing
end if
if ( not wc is Nothing ) then
wc.Dispose()
wc = Nothing
end if

end try

end sub

'----------------
'
private sub decode_base64()

dim data_array() as Byte

Trace.Warn( ">", "==============" )
Trace.Warn( ">", "decode_base64:" )

' Attempt to decode from base64 ...
try

data_array = Convert.FromBase64String( m_encoded_data )

Trace.Warn( ">", "data_array.length = " & data_array.length.ToString() )

if ( data_array.length = m_file_size ) then
Trace.Warn( ">", "DECODE OK" )
else
Trace.Warn( ">", "DECODE FAILED!" )
end if

catch ex as Exception
Trace.Warn( ">", ex.ToString() )

end try

end sub
</script>
<html>
<head>
<title>Attempt to encode/decode using Base 64</title>
</head>
<body>
<blockquote>

<form runat="server">
Enter the URL to a file, preferably a small image JPG or GIF file.
<hr noshade size="1">
<table width="90%" border="0" cellpadding="2" cellspacing="1">
<tr>
<td width="20%" align="right" valign="top" class="form_label_1">
URL:
</td>
<td width="80%" align="left" valign="top" class="form_field_1">
<asp:TextBox id="fld_url" text="" size="100" maxlength="200" runat="server" />
</td>
</tr>
</table>
<hr noshade size="1">
<table width="90%" border="0" cellpadding="2" cellspacing="1">
<tr>
<td width="20%"> </td>
<td width="80%" valign="top">
<asp:Button OnClick="do_test" text="Do Base64 Test" runat="server" />
</td>
</tr>
</table>
</form>

</blockquote>
</body>
</html>