System.Net.WebClient

Hey all... i have to login a page and upload a file to that page ( cant
access to the code ) , Patrice and Kevin told me i could do it with
System.Net.WebClient, i could, but with a test site here in my intranet,
that website ( HTTPS ) detects i m not using the login form.. and i cant log
in that app...how can an ASP page detects if i login with the site or using
webclient ?
tnx..
CHIN@dotnet.itags.org.KD
www.racingclub.com.arGenerally the web app will look for a cookie in the request that was
issued during a succesful login (assuming the web app is using forms
authentication).
Some tips on how to programatically login:
http://odetocode.com/Articles/162.aspx
Scott
http://www.OdeToCode.com/blogs/scott/
On Tue, 1 Feb 2005 09:22:58 -0300, "CHIN@.KD"
<racingnet_ac@.yahoo.com.ar> wrote:

>Hey all... i have to login a page and upload a file to that page ( cant
>access to the code ) , Patrice and Kevin told me i could do it with
>System.Net.WebClient, i could, but with a test site here in my intranet,
>that website ( HTTPS ) detects i m not using the login form.. and i cant lo
g
>in that app...how can an ASP page detects if i login with the site or using
>webclient ?
>tnx..

System.Net.WebClient

Hey all... i have to login a page and upload a file to that page ( cant
access to the code ) , Patrice and Kevin told me i could do it with
System.Net.WebClient, i could, but with a test site here in my intranet,
that website ( HTTPS ) detects i m not using the login form.. and i cant log
in that app...how can an ASP page detects if i login with the site or using
webclient ?

tnx..

--
CHIN@dotnet.itags.org.KD
www.racingclub.com.arGenerally the web app will look for a cookie in the request that was
issued during a succesful login (assuming the web app is using forms
authentication).

Some tips on how to programatically login:
http://odetocode.com/Articles/162.aspx

--
Scott
http://www.OdeToCode.com/blogs/scott/

On Tue, 1 Feb 2005 09:22:58 -0300, "CHIN@.KD"
<racingnet_ac@.yahoo.com.ar> wrote:

>Hey all... i have to login a page and upload a file to that page ( cant
>access to the code ) , Patrice and Kevin told me i could do it with
>System.Net.WebClient, i could, but with a test site here in my intranet,
>that website ( HTTPS ) detects i m not using the login form.. and i cant log
>in that app...how can an ASP page detects if i login with the site or using
>webclient ?
>tnx..

System.Net.WebClient Best Practices

I've used the WebClient class on a few projects but I wanted to know if
anyone could point to the good resource for Best Practices with this object.
The two things I haven't seen in sample code are:
1. How to retrieve connection error messages, such as a DNS resolution error
or connection dropped errors.
2. Should I use the Dispose or Finalize methods to destroy the object when I
done? I assume that the object uses unmanaged code at some point to connect
to another server. So do I need to use the Finalize method release those
other objects?
Here is a function from one of my projects:
Private Function getRemoteDat() As Boolean
Dim CCPostURL As String =
ConfigurationSettings.AppSettings("https://www.remoteData.com/process.asp")
Dim Status As Boolean = False ' Assumes failure until set to True
Dim ResponseResults As String
Dim responseString As String
Dim myWebClient As New System.Net.WebClient
Dim responseArray As Byte()
Dim myNameValueCollection As New
System.Collections.Specialized.NameValueCollection
Dim ResultMsg As String
Dim splitResponse() As String
myNameValueCollection.Add("first_name", txtBillFName.Text)
myNameValueCollection.Add("last_name", txtBillLName.Text)
myNameValueCollection.Add("address", txtBillAdd1.Text)
Try
responseArray = myWebClient.UploadValues(CCPostURL, "POST",
myNameValueCollection) 'Sends request to Gateway
responseString = System.Text.Encoding.ASCII.GetString(responseArray)
'Converts bit array to string
splitResponse = Split(responseString, System.Environment.NewLine)
'Splits response into string array
ResultMsg = splitResponse(1).Remove(0, 19) 'Gets Result Message
Catch
Status = False 'Connection failed or response was corrupted
End Try
If ResultMsg = "APPROVED" Then
_GatewayApprovalString = splitResponse(2) & "|" & splitResponse(3)
'Set string if successful
Status = True
Else
Status = False
End If
Return Status
End FunctionHi jmh,
Welcome to ASP.NET newsgroup.
Regarding on the "System.Net.WebClient Best Practices" you mentioned, here
are some of my suggestions:
The WebClient class in system.net namespace is a well-encapsulated class
which just simulate the IE browser's behavior. When we use it to post data
or retrieve response from remote url, it will help do the underlying
connection and transfering tasks for us(also some additinal works such as
cookie management). In fact, the WebClient class is using the
HttpWebRequest class internally when dealing with http... resource. So if
we need more detailed and
underlying control when doing such work, we can consider directly use the
HttpWebRequest class instead. This class can help us manually write binary
data into the http request's stream and customize the http Message's header
Also, we can get the http status code after we make request and retieve
response from the remote resource. For detailed reference on
HttpWebRequest , you can lookup the MSDN document on it(or the accessing
internet section in "programming with .net section).
In addition, since httpWebrequest have more underlying controls, it also
require us to do more works to properly handle the resource. for example,
we need to manually close the response stream after read it so as not to
occupy the connection. Here are some tech articles and former newsgroup
threads discussing on such problems, such as use httpwebrequest to post
data, upload file or upload files together with form datas:
#Send data from non browser client (using HttpWebRequest) that can be used
by access web page controls.
http://weblogs.asp.net/ngur/archive.../11/129951.aspx
#how to upload file via c# code
http://groups-beta.google.com/group...anguages.csharp
/browse_thread/thread/eea92e16a26fbdc0/474b6a3fcd63dd93?q=C%23+upload+file+s
teven+cheng&rnum=1&hl=en#474b6a3fcd63dd93
Steven Cheng
Microsoft Online Support
Get Secure! www.microsoft.com/security
(This posting is provided "AS IS", with no warranties, and confers no
rights.)

System.Net.WebClient Best Practices

I've used the WebClient class on a few projects but I wanted to know if
anyone could point to the good resource for Best Practices with this object.

The two things I haven't seen in sample code are:
1. How to retrieve connection error messages, such as a DNS resolution error
or connection dropped errors.
2. Should I use the Dispose or Finalize methods to destroy the object when I
done? I assume that the object uses unmanaged code at some point to connect
to another server. So do I need to use the Finalize method release those
other objects?

Here is a function from one of my projects:

Private Function getRemoteDat() As Boolean
Dim CCPostURL As String =
ConfigurationSettings.AppSettings("https://www.remoteData.com/process.asp")

Dim Status As Boolean = False ' Assumes failure until set to True
Dim ResponseResults As String
Dim responseString As String
Dim myWebClient As New System.Net.WebClient
Dim responseArray As Byte()
Dim myNameValueCollection As New
System.Collections.Specialized.NameValueCollection
Dim ResultMsg As String
Dim splitResponse() As String

myNameValueCollection.Add("first_name", txtBillFName.Text)
myNameValueCollection.Add("last_name", txtBillLName.Text)
myNameValueCollection.Add("address", txtBillAdd1.Text)

Try
responseArray = myWebClient.UploadValues(CCPostURL, "POST",
myNameValueCollection) 'Sends request to Gateway
responseString = System.Text.Encoding.ASCII.GetString(responseArray )
'Converts bit array to string
splitResponse = Split(responseString, System.Environment.NewLine)
'Splits response into string array
ResultMsg = splitResponse(1).Remove(0, 19) 'Gets Result Message
Catch
Status = False 'Connection failed or response was corrupted
End Try

If ResultMsg = "APPROVED" Then
_GatewayApprovalString = splitResponse(2) & "|" & splitResponse(3)
'Set string if successful
Status = True
Else
Status = False
End If

Return Status
End FunctionHi jmh,

Welcome to ASP.NET newsgroup.
Regarding on the "System.Net.WebClient Best Practices" you mentioned, here
are some of my suggestions:

The WebClient class in system.net namespace is a well-encapsulated class
which just simulate the IE browser's behavior. When we use it to post data
or retrieve response from remote url, it will help do the underlying
connection and transfering tasks for us(also some additinal works such as
cookie management). In fact, the WebClient class is using the
HttpWebRequest class internally when dealing with http... resource. So if
we need more detailed and
underlying control when doing such work, we can consider directly use the
HttpWebRequest class instead. This class can help us manually write binary
data into the http request's stream and customize the http Message's header
Also, we can get the http status code after we make request and retieve
response from the remote resource. For detailed reference on
HttpWebRequest , you can lookup the MSDN document on it(or the accessing
internet section in "programming with .net section).

In addition, since httpWebrequest have more underlying controls, it also
require us to do more works to properly handle the resource. for example,
we need to manually close the response stream after read it so as not to
occupy the connection. Here are some tech articles and former newsgroup
threads discussing on such problems, such as use httpwebrequest to post
data, upload file or upload files together with form datas:

#Send data from non browser client (using HttpWebRequest) that can be used
by access web page controls.
http://weblogs.asp.net/ngur/archive.../11/129951.aspx

#how to upload file via c# code
http://groups-beta.google.com/group...anguages.csharp
/browse_thread/thread/eea92e16a26fbdc0/474b6a3fcd63dd93?q=C%23+upload+file+s
teven+cheng&rnum=1&hl=en#474b6a3fcd63dd93

Steven Cheng
Microsoft Online Support

Get Secure! www.microsoft.com/security
(This posting is provided "AS IS", with no warranties, and confers no
rights.)

System.Net.WebClient and UploadFile() Problem

I am using WebClient to upload a file to a remote server. The call I'm
making looks as follows:
webClient.UploadFile("http://sdtpcal.someserver.com/Federation/Databases/Fed
eration.Zip",
@dotnet.itags.org."C:\Temp\Federation.Zip");
'Federation' is a virtual directory on the server I'm attempting to upload
the file to. The URL is valid and the file I'm attempting to transfer
exists. I keep getting 404 errors (not found). It's not clear to me exactly
what cannot be found. If I attempt to download this exact file from the same
exact URL using webClient.DownloadFile, it works fine!
Thanks for the help - Amos.You can not simply upload file on a server.
Something should wait for that file at the server end.
So url http://sdtpcal.someserver.com/Feder.../Federation.Zip to
download file looks correct.
but to upload you must write an ASPX page that will accept file and save to
the folder on a server.
George.
"Amos Soma" <amos_j_soma@.yahoo.com> wrote in message
news:OqGt9oK7GHA.1248@.TK2MSFTNGP03.phx.gbl...
>I am using WebClient to upload a file to a remote server. The call I'm
>making looks as follows:
> webClient.UploadFile("http://sdtpcal.someserver.com/Federation/Databases/F
ederation.Zip",
> @."C:\Temp\Federation.Zip");
> 'Federation' is a virtual directory on the server I'm attempting to upload
> the file to. The URL is valid and the file I'm attempting to transfer
> exists. I keep getting 404 errors (not found). It's not clear to me
> exactly what cannot be found. If I attempt to download this exact file
> from the same exact URL using webClient.DownloadFile, it works fine!
> Thanks for the help - Amos.
>
Thus wrote George Ter-Saakov,

> You can not simply upload file on a server.
> Something should wait for that file at the server end.
> So url
> http://sdtpcal.someserver.com/Feder.../Federation.Zip to
> download file looks correct.
> but to upload you must write an ASPX page that will accept file and
> save to the folder on a server.
You only need a web application endpoint for POST requests. If the OP can
use PUT, the web server may support this without any special user code (IIS
does for sure).
Both WebClient.UploadFile(uri, "PUT", fileName) and WebClient.OpenWrite(uri,
"PUT") will do the trick.
Cheers,
--
Joerg Jooss
news-reply@.joergjooss.de

System.Net.WebClient and UploadFile() Problem

I am using WebClient to upload a file to a remote server. The call I'm
making looks as follows:

webClient.UploadFile("http://sdtpcal.someserver.com/Federation/Databases/Federation.Zip",
@dotnet.itags.org."C:\Temp\Federation.Zip");

'Federation' is a virtual directory on the server I'm attempting to upload
the file to. The URL is valid and the file I'm attempting to transfer
exists. I keep getting 404 errors (not found). It's not clear to me exactly
what cannot be found. If I attempt to download this exact file from the same
exact URL using webClient.DownloadFile, it works fine!

Thanks for the help - Amos.You can not simply upload file on a server.
Something should wait for that file at the server end.

So url http://sdtpcal.someserver.com/Feder.../Federation.Zip to
download file looks correct.

but to upload you must write an ASPX page that will accept file and save to
the folder on a server.

George.

"Amos Soma" <amos_j_soma@.yahoo.comwrote in message
news:OqGt9oK7GHA.1248@.TK2MSFTNGP03.phx.gbl...

Quote:

Originally Posted by

>I am using WebClient to upload a file to a remote server. The call I'm
>making looks as follows:
>
webClient.UploadFile("http://sdtpcal.someserver.com/Federation/Databases/Federation.Zip",
@."C:\Temp\Federation.Zip");
>
'Federation' is a virtual directory on the server I'm attempting to upload
the file to. The URL is valid and the file I'm attempting to transfer
exists. I keep getting 404 errors (not found). It's not clear to me
exactly what cannot be found. If I attempt to download this exact file
from the same exact URL using webClient.DownloadFile, it works fine!
>
Thanks for the help - Amos.
>
>

Thus wrote George Ter-Saakov,

Quote:

Originally Posted by

You can not simply upload file on a server.
Something should wait for that file at the server end.
So url
http://sdtpcal.someserver.com/Feder.../Federation.Zip to
download file looks correct.
>
but to upload you must write an ASPX page that will accept file and
save to the folder on a server.

You only need a web application endpoint for POST requests. If the OP can
use PUT, the web server may support this without any special user code (IIS
does for sure).

Both WebClient.UploadFile(uri, "PUT", fileName) and WebClient.OpenWrite(uri,
"PUT") will do the trick.

Cheers,
--
Joerg Jooss
news-reply@.joergjooss.de

System.Net.Webclient screen scraping: how to gracefully handle 403 (and other) errors?

I've written a very small ASP.NET page to scrape thousands of pages of
content based on database IDs. It loops through a dataset to get the IDs. It
worked well in testing but now I am getting an annoying 403 error that
causes the script to abort halfway through my download.

I am wondering if there is a way in ASP.NET to have my code ignore 403
errors and other network errors, catch the error, and iterate to the next ID
in the dataset rather than aborting the whole job.

My code appears below. Thank you in advance.

-KF

string strConnection;

strConnection = ConfigurationSettings.AppSettings["connwhatever"];

SqlConnection conn = new SqlConnection(strConnection);

string query = // [my query];

SqlDataAdapter a = new SqlDataAdapter(query, conn);

DataSet s = new DataSet();

a.Fill(s);

int counter = 0;

foreach (DataRow dr in s.Tables[0].Rows)

{

counter++;

System.Net.WebClient wc = new WebClient();

string strData =
wc.DownloadString("http://whatever.org/article.asp?articleid=" +
dr[0].ToString());

FileStream fstream = new FileStream(@dotnet.itags.org."c:\whateverpath\" + dr[0].ToString() +
".htm", FileMode.Create, FileAccess.Write);

StreamWriter stream = new StreamWriter(fstream);

stream.Write(strData);

stream.Close();

fstream.Close();please read chapter on try/catch

-- bruce (sqlwork.com)

kenfine@.nospam.nospam wrote:

Quote:

Originally Posted by

I've written a very small ASP.NET page to scrape thousands of pages of
content based on database IDs. It loops through a dataset to get the IDs. It
worked well in testing but now I am getting an annoying 403 error that
causes the script to abort halfway through my download.
>
I am wondering if there is a way in ASP.NET to have my code ignore 403
errors and other network errors, catch the error, and iterate to the next ID
in the dataset rather than aborting the whole job.
>
My code appears below. Thank you in advance.
>
-KF
>
string strConnection;
>
strConnection = ConfigurationSettings.AppSettings["connwhatever"];
>
SqlConnection conn = new SqlConnection(strConnection);
>
string query = // [my query];
>
SqlDataAdapter a = new SqlDataAdapter(query, conn);
>
DataSet s = new DataSet();
>
a.Fill(s);
>
int counter = 0;
>
foreach (DataRow dr in s.Tables[0].Rows)
>
{
>
counter++;
>
System.Net.WebClient wc = new WebClient();
>
string strData =
wc.DownloadString("http://whatever.org/article.asp?articleid=" +
dr[0].ToString());
>
FileStream fstream = new FileStream(@."c:\whateverpath\" + dr[0].ToString() +
".htm", FileMode.Create, FileAccess.Write);
>
StreamWriter stream = new StreamWriter(fstream);
>
stream.Write(strData);
>
stream.Close();
>
fstream.Close();
>
>
>
>

Understand try/catch generally. What event(s) should I be trying to catch?

Thank you,
-KF

"bruce barker" <nospam@.nospam.comwrote in message
news:%23F79bSINHHA.3288@.TK2MSFTNGP03.phx.gbl...

Quote:

Originally Posted by

please read chapter on try/catch
>
-- bruce (sqlwork.com)
>
kenfine@.nospam.nospam wrote:

Quote:

Originally Posted by

>I've written a very small ASP.NET page to scrape thousands of pages of
>content based on database IDs. It loops through a dataset to get the IDs.
>It worked well in testing but now I am getting an annoying 403 error that
>causes the script to abort halfway through my download.
>>
>I am wondering if there is a way in ASP.NET to have my code ignore 403
>errors and other network errors, catch the error, and iterate to the next
>ID in the dataset rather than aborting the whole job.
>>
>My code appears below. Thank you in advance.
>>
>-KF
>>
>string strConnection;
>>
>strConnection = ConfigurationSettings.AppSettings["connwhatever"];
>>
>SqlConnection conn = new SqlConnection(strConnection);
>>
>string query = // [my query];
>>
>SqlDataAdapter a = new SqlDataAdapter(query, conn);
>>
>DataSet s = new DataSet();
>>
>a.Fill(s);
>>
>int counter = 0;
>>
>foreach (DataRow dr in s.Tables[0].Rows)
>>
>{
>>
>counter++;
>>
>System.Net.WebClient wc = new WebClient();
>>
>string strData =
>wc.DownloadString("http://whatever.org/article.asp?articleid=" +
>dr[0].ToString());
>>
>FileStream fstream = new FileStream(@."c:\whateverpath\" +
>dr[0].ToString() + ".htm", FileMode.Create, FileAccess.Write);
>>
>StreamWriter stream = new StreamWriter(fstream);
>>
>stream.Write(strData);
>>
>stream.Close();
>>
>fstream.Close();
>>
>>
>>

Hello KF,

Based on your description, you're using the webclient class to request many
web pages programmatically in ASP.NET page code. However, since some page
may raise some exception, your client loop code in ASP.NET page break,
correct?

As for the 403 error, it is normally caused by the security authorization
checking at server-side fails. I'm not sure whether there is any other
particular scenario here, however, if what you want is simply captuer and
ignore such error and continue the loop, you can just add a try catch block
around your webclient class's downloadXXX method call and if any exception
captured you can simply ignore it and skip the current loop. e.g.

=======================
foreach (DataRow dr in s.Tables[0].Rows)

{

counter++;

System.Net.WebClient wc = new WebClient();

try
{

string strData =
wc.DownloadString("http://whatever.org/article.asp?articleid=" +
dr[0].ToString());

}catch(Exception ex)
{
//ignore and continue the loop
}

...........................

}
=========================

Does this work for your scenario?

Sincerely,

Steven Cheng

Microsoft MSDN Online Support Lead

==================================================

Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/subscript...ault.aspx#notif
ications.

Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscript...rt/default.aspx.

==================================================

This posting is provided "AS IS" with no warranties, and confers no rights.
For webclient or HttpWebRequest, it normally will throw a
System.Net.WebException, however, any exception can be handled by the super
class "Exception". So you can use either

try
{
}catch(Exception)
{

}

or

try
{
}catch(WebException)
{

}

Sincerely,

Steven Cheng

Microsoft MSDN Online Support Lead

This posting is provided "AS IS" with no warranties, and confers no rights.
You can specify error documents for specific error-codes.

In your web.config, add the following entries in <system.websection:

<customErrors>
<error statusCode="403" redirect="403.aspx"/>
</customErrors>

Note that generally, 403 would be given by the web server and not be the
ASP.Net engine. At times, when the authentication fails, 403 may be returned
by an IHttpModule - like the authentication modules (NTML, Kerberos, Digest
etc).

--
Happy Hacking,
Gaurav Vaish | www.mastergaurav.com
www.edujini-labs.com
http://eduzine.edujinionline.com
-------------

<kenfine@.nospam.nospamwrote in message
news:%23SginPGNHHA.4916@.TK2MSFTNGP06.phx.gbl...

Quote:

Originally Posted by

I've written a very small ASP.NET page to scrape thousands of pages of
content based on database IDs. It loops through a dataset to get the IDs.
It worked well in testing but now I am getting an annoying 403 error that
causes the script to abort halfway through my download.
>
I am wondering if there is a way in ASP.NET to have my code ignore 403
errors and other network errors, catch the error, and iterate to the next
ID in the dataset rather than aborting the whole job.
>
My code appears below. Thank you in advance.

This worked great for my scenario. Thanks very much to everyone for the
timely assistance.

-KF

"Steven Cheng[MSFT]" <stcheng@.online.microsoft.comwrote in message
news:FYKH6FJNHHA.2024@.TK2MSFTNGHUB02.phx.gbl...

Quote:

Originally Posted by

Hello KF,
>
Based on your description, you're using the webclient class to request
many
web pages programmatically in ASP.NET page code. However, since some page
may raise some exception, your client loop code in ASP.NET page break,
correct?
>
As for the 403 error, it is normally caused by the security authorization
checking at server-side fails. I'm not sure whether there is any other
particular scenario here, however, if what you want is simply captuer and
ignore such error and continue the loop, you can just add a try catch
block
around your webclient class's downloadXXX method call and if any exception
captured you can simply ignore it and skip the current loop. e.g.
>
=======================
foreach (DataRow dr in s.Tables[0].Rows)
>
{
>
counter++;
>
System.Net.WebClient wc = new WebClient();
>
try
{
>
string strData =
wc.DownloadString("http://whatever.org/article.asp?articleid=" +
dr[0].ToString());
>
}catch(Exception ex)
{
//ignore and continue the loop
}
>
...........................
>
}
=========================
>
Does this work for your scenario?
>
Sincerely,
>
Steven Cheng
>
Microsoft MSDN Online Support Lead
>
>
>
==================================================
>
Get notification to my posts through email? Please refer to
http://msdn.microsoft.com/subscript...ault.aspx#notif
ications.
>
>
>
Note: The MSDN Managed Newsgroup support offering is for non-urgent issues
where an initial response from the community or a Microsoft Support
Engineer within 1 business day is acceptable. Please note that each follow
up response may take approximately 2 business days as the support
professional working with you may need further investigation to reach the
most efficient resolution. The offering is not appropriate for situations
that require urgent, real-time or phone-based interactions or complex
project analysis and dump analysis issues. Issues of this nature are best
handled working with a dedicated Microsoft Support Engineer by contacting
Microsoft Customer Support Services (CSS) at
http://msdn.microsoft.com/subscript...rt/default.aspx.
>
==================================================
>
>
>
This posting is provided "AS IS" with no warranties, and confers no
rights.
>

System.Net.Webclient screen scraping: how to gracefully handle 403 (and other) er

Understand try/catch generally. What event(s) should I be trying to catch?
Thank you,
-KF
"bruce barker" <nospam@dotnet.itags.org.nospam.com> wrote in message
news:%23F79bSINHHA.3288@dotnet.itags.org.TK2MSFTNGP03.phx.gbl...
> please read chapter on try/catch
> -- bruce (sqlwork.com)
> kenfine@dotnet.itags.org.nospam.nospam wrote:For webclient or HttpWebRequest, it normally will throw a
System.Net.WebException, however, any exception can be handled by the super
class "Exception". So you can use either
try
{
}catch(Exception)
{
}
or
try
{
}catch(WebException)
{
}
Sincerely,
Steven Cheng
Microsoft MSDN Online Support Lead
This posting is provided "AS IS" with no warranties, and confers no rights.