CHAPTER SIX - CGIs
"cgi" stands for "common gateway interface", a fancy name meaning computer programs running on the web server that can be invoked from a WWW page at the browser. The "bin" part alludes to the binary executables that result from compiled or assembled programs. It is a bit misleading because cgi's can also be Unix shell scripts or interpreted languages like Perl.
A typical use for cgi is the processing of online forms. When the user fills in the boxes on the form and hits the SUBMIT button, the cgi program specified in the html will be run at the server, and the information in the boxes become available to the program as parameters. The program, being a program, can then do anything the programmer wanted it to do.
"cgiemail", for example, is a canned program written in C that gathers up the contents of the boxes on the form and emails them to a specified destination, then sends a WWW page confirming the action.
Your server will determine if a file is a CGI by the file extension. ALL CGI SCRIPTS MUST END WITH THE EXTENSION .CGI - e.g., form.cgi . For example, /www/htdocs/company/cgi-local/script.cgi . Put your cgi scripts anywhere - although we suggest a directory called anything but cgi-bin - since that is where the system scripts go!
If your domain is named company.com, you would then access your scripts as http://www.company.com/cgi-local/script.cgi.
NOTE: not http://www.company.com/cgi-bin/script. No, you can't make a subdirectory of /www/htdocs/company/cgi-bin and run programs from there.
To find the path of the program date, type which date. Do this for any program that you need your cgi-bin script to access.
I see lots of perls, which one should I use?
They should all be the same. /usr/local/bin/perl should work nicely.
The installation of a CGI script often requires a special file permission; however, please note that the appropriate permissions are set automatically upon upload to your cgi-local. However, in some instances it may still be necessary to set a file or directory permission.
Our recommended FTP client, WS_FTP, can be used to set any numeric chmod permission setting.
Suppose you were preparing to enable anonymous FTP and you needed to disable public write access to a file named sample.htm. As mentioned, the File Manager can be used to accomplish this task, but for this example, the WSFTP SITE command is used.
The command to set public write access
OFF is: chmod 750 <filename>
Follow the instructions above to display the SITE command field and enter the desired chmod command as shown here.
If the file in question resides in a subdirectory of your web site, you will first need to open the respective subdirectory before issuing the SITE command.
So what does the "750" mean? The 750 number tells chmod to make the "sample.htm"file readable, writable, and executable by the owner of the file (you), readable and executable by the group of the file while making it neither readable, writable, nor executable by the world. How did we come up with the number? Well, files in UNIX have three types of permissions: USER (the owner of the file), GROUP (The security group you are in), and OTHER (For the world to see. Each digit in the number above corresponds to one of these categories. The first digit is user, the second digit is group, and the final digit is other. Thus, in the example above, 7=USER, 5=GROUP and OTHER=0. The actual value of the digit determines the permissions granted to that area. Permissions consist of three numbers -- 4 for read, 2 for write, and 1 for execute access. By adding these numbers together, you form the permissions that make up one digit. For example, 4 + 2 + 1 = 7 which grants read, write, and execute permissions. 4 + 1 = 5 which only grants read and execute permissions. Thus, 755 grants 7 (read, write, execute) to the owner of the file, and 5 (read and execute) to the group the file is in and the world. Below is a chart which will can be used as a quick reference
PERMISSION COMMAND U G W rwx rwx rwx chmod 777 filename rwx rwx r-x chmod 775 filename rwx r-x r-x chmod 755 filename rw- rw- r-- chmod 664 filename rw- r-- r-- chmod 644 filename U = User G = Group W = World r = Readable w = writable x = executable - = no permission
[WARNING] You may be tempted to simply use chmod 777 on all the files and directories since that assures the Web server can do anything with the files. However, it is strongly advised that you do not leave the files in this state. It is considered a big security risk to leave your scripts open to changes by the Web server instead of being read-only. Anyone on the server could use another rogue CGI script to write over your scripts and make them do something completely different. There is still a risk involved in making the messages directory writable, but at least if someone is going to be messing with your area, they will only destroy a bit of data and not your main programs. It is "OK" to set the scripts to 777 if you are troubleshooting a problem and want to rule out permissions entirely, but do not leave the scripts like this. On another security note, if you are really concerned with the security of your data such as, please do not use a shared server where other people can write CGI scripts using the same Web server configuration. It is much better to use your own server software or purchase space on a "virtual server" which may be shared, but is set up in such a way that each user's scripts are shielded from each other.
Note: Not setting your permissions correctly is the NUMBER 1 reason why script installations fail. Take time to get this right.
Cgiemail is a program written in the C language that takes the contents of fill-in boxes on a form and emails them to a specified destination. In addition to the form specification in the .html file, a mail specification in a .txt file is required to format the resulting email message.
We provide cgiemail in the cgi-bin directory of each machine. Thus, it would be called with a URL such as /cgi-bin/cgiemail/mail.txt. Details are provided below.
Please understand that cgiemail will NOT be in your cgi-bin directory. We store it in the main cgi-bin directory of each machine. The instructions given will allow you to use it without copying it to your cgi-bin directory.
While there are a number of subsections below this one, they all work together and are meant to be read from start to finish.
When developing The List, having a good form to present to the user for adding new providers was essential. Without this, it would have almost never gotten all the information needed, and the developers of this site would have spent five times as much time reformatting the message before putting it in the database. Thus, we will use an old version of the "add" form from that site to demonstrate cgiemail.
You will want to either look at the form before the discussion begins, or perhaps open another window to keep it on-screen. (Netscape Navigator users, right-click on the link to open it in another window.)
If you've never dealt with HTML forms before, don't worry. They're easy to create and understand. (And if you briefly tried that link and thought, noooooo!!! I don't want to read this!, scroll down a bit. The easy examples are at the bottom. You should not need to follow it at all to understand what will be explained presently.)
The form prompts the user for data which is sent to the server as simple key-value pairs. Each <input> tag specifies a record. The key is given by the name attribute, and the value is given by the value attribute. The type attribute tells the browser what kind of data to expect. Now, try looking at the example.
Please note that the hidden items are used to transmit critical info to cgiemail. They provide the location of the success file, the name of the person the results should be sent to, and the subject of the form. When making your own forms, you would definitely (please!) change the e-mail address in the "required-to" field, and likely the subject in the "subject" field. The first item tells cgiemail what to show the user after succesfully completing the form. You can, but don't need to, customize this.
After that come the items that are actually presented to the user. You'll want to use type=text input items with cgiemail; it's a simple tool. The size=60 tells the browser how big to make the box. The name=something is required in each input tag, otherwise the browser wouldn't know how to send the data to the server. The value="" attribute is correct in most cases, unless you want a default value in the form.
Note that if a field begins with required-, cgiemail will require that the user enter a value for this field. This is particularly useful if you want to require a user to submit their e-mail address, name, or something else essential.
When the user presses the Submit button, the data goes to our machine where cgiemail starts doing something with it. What it does is controlled by ...
Make sure that you upload mail.txt in ASCII mode.
Failure to upload mail.txt in ASCII mode will generate the message:
Server Error The server encountered an internal error or misconfiguration and was unable to complete your request.
Now that we have all this data, what do we do with it? Mail it, of course! But for flexibility, cgiemail requires that you create a mail.txt file to show it what to send. (If you didn't want flexibility you'd use a mailto link.) The program will read in this file, perform substitutions, and pass it to the mail system.
First, how does cgiemail know where to find this file? Look at the first line in the example form used in the previous section. The bit after the word cgiemail in the action attribute tells cgiemail where to look. When called from www.clubct.com, cgiemail does not know what your domain is because it is installed for everyone. So, the first component of this path must be your domain. After that, it goes to your www directory and looks from there. If you have a domain name, we suggest that you explicitly put it in the action attribute, because the behavior is different: cgiemail does know what your domain is if called for your domain.
Snoopy.com might want to put the mail.txt file in his www directory. His form would then start off with
<form method=post action="http://www.snoopy.com/cgi-bin/cgiemail/mail.txt">
There's nothing magical about the name mail.txt. Feel free to call it mail1.txt, or form1.mail, or what suits you, as long as the form has the correct name for what you uploaded.
Here's the mail.txt we used with the form presented earlier. It's pretty simple. Note that the first several lines are mail headers. You probably shouldn't change that part, or the corresponding parts in your form. In particular, there must be a To: header or the mail won't go anywhere!
What cgiemail does is simply replace every string that looks like [key] with the value the user typed in to the field with name=key. That's all. You can lay out your form as is best for your users, but lay out your mail.txt as is best for you to read. You can even insert gobs of text to help format the output. Only the [key] parts will be replaced by cgiemail.
Actually, if you've read the previous two sections you should know enough to use it. We'll summarize here.
Here's the finished example form created by the sample.
Cgiemail was developed at MIT and is copyright 1994, 1995.
Formmail is similar to CGIMAIL with the exception that it is not necessary to have a mail.txt file. Instead, the recipient and the subject must be included as "hidden" types with the form itself.
For instance, if you were to use formmail.pl, the beginning of your form would look like this :
<form method=post action="http://www.your-name.com/cgi-bin/formmail.pl"> <input type="hidden" name="recipient" value="your-name@your-domain.com" > <input type="hidden" name="subject" value="your-subject" >
There is only one form field that you must have in your form, for FormMail to work correctly. This is the recipient field. This form field allows you to specify to whom you wish for your form results to be mailed. Most likely you will want to configure this option as a hidden form field with a value equal to that of your e-mail address as shown above.
All other form fields are optional. The subject field will allow you to specify the subject that you wish to appear in the e-mail that is sent to you after this form has been filled out. If you do not have this option turned on, then the script will default to a message subject: WWW Form Submission. To allow the user to choose a subject:
<input type=text name="subject">
The e-mail form field will allow the user to specify their return e-mail address. If you want to be able to return e-mail to your user, I strongly suggest that you include this form field and allow them to fill it in. This will be put into the From: field of the message you receive.
<input type=text name="email">
If you wish to redirect the user to a different URL, rather than having them see the default response to the fill-out form, you can use this hidden variable to send them to a pre-made HTML page. To choose the URL they will end up at:
<input type=hidden name="redirect" value="http://your.address/to/file.html">
To allow them to specify a URL they wish to travel to once the form is filled out:
<input type=text name="redirect">
You can require for certain fields in your form to be filled in before the user can successfully submit the form. Simply place all field names that you want to be mandatory into this field. If the required fields are not filled in, the user will be notified of what they need to fill in, and a link back to the form they just submitted will be provided.
If you want to require that they fill in the email and phone fields in your form, so that you can reach them once you have received the mail, use syntax like:
<input type=hidden name="required" value="email,phone">
Any other form fields that appear in your script will be mailed back to you and displayed on the resulting page if you do not have the redirect field set. There is no limit as to how many other form fields you can use with this form, except the limits imposed by browsers and your server. FormMail was written by Matt Wright and is copyright 1995.
'Imagemap' refers to an interaction between a user's browser and the WWW server which allows clicking on an image to select hyperlinks. (Other implementations have a seperate /cgi-bin/imagemap program. Our server handles them internally. Use the reference style shown below, not one like /cgi-bin/imagemap/mymap.map.) Each designated portion of the image may select different links.
Images that are to be used as imagemaps must be designated as "ISMAP". A companion .map file specifies the coordinates of the portions of the image that are hyperlinks, and the URL or local reference of the link.
The extension .map is special; so be sure that's what your map description files end in. The .map file is a text file, so be sure to upload it in ASCII (text) mode.
The following HTML:
<A HREF="name-of-map-spec-file.map"> <IMG BORDER SRC="name-of-image.gif" ISMAP></A>
tell the browser that "name-of-image.gif" is to be displayed and used as an imagemap. The "name-of-map-spec-file.map" file contains the coordinates of the clickable areas in the image and specify the hyperlinks associated with each clickable area.
The map specification file MUST have extension ".map". While this example assumes the file is located in your www directory, it may be located anywhere under the www directory. In such a case, the path from the www directory must be specified. If you put it in /home/your-id/www/map-directory, then use <A HREF="../map-directory/name-of-map-spec-file.map">
Please use the NCSA format for your .map file.
A file named imagemap.conf does NOT have to be edited when using our imagemap program. That complication has been removed from our version of imagemap.
Here is a sample file named fo-abc.map:
default http://oceania.org/weblinks.html poly http://oceania.org/weblinks.html#A 23,14 44,2 66,14 66,39 45,51 23,39 poly http://oceania.org/weblinks.html#A 67,14 88,2 110,14 110,39 89,51 67,39 poly http://oceania.org/weblinks.html#C 111,14 132,2 154,14 154,39 133,51 111,39
Note that the URL's are full URLs. You can't use relative URLs.
The number pairs specify the X-Y coordinates in pixels of the points of the three hexagons that are the clickable areas in fo-abc.map. Clicks outside of the areas specified get the default link.
The fastest way to implement a counter is to use
<img src=http://yourdomain.com/cgi-bin/nph-count?width=7&link=http://yourdomain.com/anypage>.
Note that the URL's are full URLs. You can't use relative URLs.
In the example width=7 indicates that the counter will have 7 digits (put whatever you like), and the counter will display the access record for the page which is referenced by
&link=http://yourdomain.com/anypage.
If you want the counter to appear to be ordinary text, look at index.cgi, which you can also use to update a counter on your home page. Put this file in your main web directory; you don't need to put it in a special cgi-bin directory. It must, however, be called index.cgi to have the desired effect. If you are shift-clicking on the link to download the file to your machine, rename it in the dialog box your browser should present to you.
If your primary audience is not North America, you may prefer to retrieve a European version of index.cgi, with commas and periods reversed. (Aside from a comment, the change is just two characters, BTW.)
After activating index.cgi as explained in the previous paragraph, create a file named .count with one line that contains the number 0. Then type chmod o+w .count to allow the counter to be updated. (chmod 666 .count if you're stuck with FTP.) The counter and index.cgi are in your main page in this example.
When http://your-domain.com is referenced at the browser, instead of delivering the file index.html as normally expected, the program index.cgi is run instead. index.cgi reads the .count file, increments the number in that file by one, and writes the file back on disk. Then it reads in the index.html file, finds the string "#COUNT" imbedded somewhere in it, changes "#COUNT" to 1 + whatever value if found in .count, and sends the modified index.html to the browser.
EASY STEPS FOR CREATING A COUNTER USING TELNET :
To create a counter on a page other than your home page, change the source from index.html
to the page that you wish to update. You would want to make another copy of index.cgi as
well. So for example, if you wished to count a page called orders.html, you would change
the source from index.html to orders.html, change the name from index.cgi to orders.cgi
and then call this page from other pages by the name orders.cgi. Change two things within
orders.cgi: find the string 'index.html' and change it to 'orders.html', and find the
string '.count' and change it to '.countorders'. Both are right near the top of the
program. Then make a file .countorders instead of .count, but otherwise as directed above.
CGIWrap is a program that allows scripts to be run by the Web Server so that it has the permissions of the owner of the script. In other words, if the Web Server is running a script owned by you, it can assume your priviledges and do such things as write to files and directories owned by you. You therefore do not need to create world-writable directories for such things as bulletin boards, guestbooks, etc.
Using CGIWrap is easy. In all of the below examples, replace SERVER with the hostname and port of your webserver. Replace USER with your userid, and replace SCRIPT with the pathname of the script that is located in the directory containing your web pages. (e.g., If you have the script count.cgi is your directory cgis, then SCRIPT=cgis/count.cgi.)
In the below URLs you can replace "cgiwrap" according to the following:
The syntax examples above using user= and script= are carryovers from the original versions of CGIwrap, and are not really needed. We highly reccomend you don't use them, the other examples are much clearer and easier to use.
(This section will grow as we remember to add what we keep doing every day anyway.)
"When I activate my CGI program, I get back a page that says 'Server Misconfigured'".
Log on and go to your cgi directory. Try running the program manually to see what errors you get. Remember that when a CGI program produces error output, that output is discarded and you get the non-specific page you saw. To check on it, you'll have to run it yourself.
If you get something like
bash: ./search.pl: No such file or directory
and you know you typed the name correctly, it's almost certain you uploaded the script from your PC or Macintosh in binary mode. Perl scripts are ASCII text and must be transferred in ASCII mode. (So are shell scripts and any other kind of script that starts with '#!' on the first line.) You know it's this if ./search.pl doesn't work and perl search.pl does. Assuming you don't have any legitimate carriage returns in the program (doubtful), this will fix it:
mv search.pl search.pl.txt && tr -d '\r' <search.pl.txt >search.pl
Then there's this response:
Literal @clubct now requires backslash at ./bigones line 16, within string
Many scripts that you'll find widely distributed are still written for Perl, version 4. Our /usr/bin/perl is version 5, which is 99.9% compatible with version 4, along with many improvements. You've found the other 0.1%.
In Perl 4, you could get away with a statement like
$mailaddress = "joe@schmoe.com";
because Perl didn't think the '@' sign was special in a string. Perl 5 does, so you need to escape it, like so:
$mailaddres = "joe\@schmoe.com";
You can't just change all the '@' signs to '\@', only the ones in strings. To speed your search, however, Perl gives you the line number of the problem line(s) in its error message. (To check for them without accidentally running the program, if that would be a problem, use perl -c script.) Fix them, and see if it works.
Tom Christianson (well-known in the Perl world) has written The Idiot's Guide to Solving Perl
CGI problems for the comp.lang.perl.misc newsgroup. Not everything in there is correct
for our ClubCt.Com LLC setup, but it's useful.