WEB4J. Because simple is beautiful.

User Guide

Table of Contents

Example Applications

• Fish & Chips Club - an intranet app. Its access control is based on roles.
• Predictions - a public web app. Its access control is per user, not per role. (Predictions is significantly simpler than Fish & Chips Club.)

Please see the Getting Started Guide for more information.

Tip - It's a good idea to modify one of the provided example apps to more closely match the needs of your organization. Then, each time you start a new project, you have a starting point which is closer to your end goal.

For those looking for a more compact introduction to WEB4J, or for those with less experience with servlets, please see the tutorial.

Why WEB4J Was Built

How WEB4J Was Built

WEB4J was not built like most tools, because it was not built with a specific implementation technique in mind. Rather, it was built with specific feelings or esthetics in mind. The basic steps were:

1. build typical applications on top of the servlet and JSP APIs
2. find all repeated code and move it from the applications into the framework
3. do it such that the application programmer will feel the maximum elegance in their code
4. do not presuppose any particular style of implementation

• Hibernate's central idea is that database schemas should be repeated in the application layer, such that SQL statements can usually be avoided, and replaced by object graph navigation
• Java Server Faces' central idea is that web application front ends are best implemented with the 'events-and-components' style of programming used in graphical user interfaces

However, it does not seem possible to produce elegance without treating it as the single most important thing. Is Hibernate beautiful? Is all that XML configuration beautiful? Are its abundant cyclic references between packages beautiful? Is needing to configure the database schema in the application layer an elegant thing? Many would say "No, this is a bit ugly". And perhaps it was destined to be ugly, since it was based on an implementation idea, instead of on esthetic principles. If you wish to make something beautiful, you need to treat beauty as your Prime Directive.

"Great software requires a fanatical devotion to beauty." - Paul Graham

Tour of a Typical Feature

Here are some typical line counts per feature, taken from a cross section of several features in the Fish & Chips Club example application. (Documentation comments are included in the line counts.)

Item	Avg Lines	Relative Size
SQL file (.sql)	25
Presentation (.jsp)	108
Total	133
Model (.java)	111
Action (.java)	144
DAO (.java)	46
Total	301

In general, your code will only be roughly twice the size of the non-code elements of your application.

It's often the case that the DAO class is simple, consisting only of a few single-line methods. In such cases, it's recommended that those DAO methods be refactored into the Action class. This will slightly reduce the total line count.

The above diagram shows the relationships between the elements found in a typical feature. The above notation means, for example, that the DAO uses the Model (and not the reverse). The View here is usually a Java Server Page.

Directory Structure

There are numerous advantages to this style. Although it may be unfamiliar to you, it's very likely that once you use it, you will find it superior to other styles.

If you do select this style, then you may find it convenient to use fixed, conventional names for both the view (the JSP) and the .sql file (which contains SQL statements), such as view.jsp and statements.sql. Using fixed names for the java classes is not recommended, however, since that would cause problems with javadoc, and would likely make your IDE less effective.

It's useful to treat your Action classes as the public face of each feature. That is, the javadoc of the Action should be the entry point into the rest of the implementation. In particular, it should link to the actual SQL statements and JSP. This is implememented in the example apps using two custom taglets. (Taglets are part of the javadoc tool.)

Example of their use:

/**
* Edit Resto objects.
*
* @sql statements.sql
* @view view.jsp
*/
public final class RestoAction ...

Model Objects

• a constructor with arguments for all possible data
• various getXXX methods
• the toString, equals, and hashCode methods

• Resto
• Member
• Visit

The primary job of a Model Object is to carry and validate data. Validation is especially important, since data is king. Most of the code in your Model Objects will deal with validation.

In WEB4J, validation is always placed in the constructor. If a validation fails, then a checked ModelCtorException must be thrown. When a validation fails, you cannot throw any other kind of exception except ModelCtorException. For example, if a required field is found to be null, then you cannot throw a NullPointerException. You must throw a ModelCtorException.

Some may object to this policy. However, you must remember that NullPointerException is an unchecked exception. Unchecked exceptions are strictly meant for bugs. Here's the important point: faulty user input is not a bug. It's to be expected as part of the normal operation of the program.

Validator and Check can help you validate your Model Objects. You are not required to use them, but you will likely find them convenient. The Check class returns common implementations of the Validator interface. For validations specific to your problem domain, you can define them as a new Validator, and then use them with the Check class, just like any other Validator.

Recommendation - Although it's not required, it's strongly recommended that your Model Objects be implemented as immutable objects. Implementing them as JavaBeans is not recommended, because of the problems linked to JavaBeans. If some other tool happens to demand that your Model Object also be a JavaBean, then your Model Object can certainly add the required members, if needed, but WEB4J itself will never require them.

To always implement Model Objects safely, you need to understand that mutable object fields often need special care, in the form of a defensive copy.

The ModelUtil class is provided to help you correctly implement the toString, equals, and hashCode methods.

Your Model Objects can use the SafeText, Decimal, Id, and DateTime classes as building blocks, in the same way as Integer, Boolean, and so on. An Id can hold any kind of identifer you wish. SafeText should usually be used to model free-form user input, to avoid issues with Cross-Site Scripting attacks (see below). The Decimal class is the recommended replacement for BigDecimal, since it makes calculations much easier. The DateTime class is the recommended replacement for the widely reviled java.util.Date.

When modeling search criteria entered into a form, it's often possible to create a Model Object with package-private getXXX methods. As a general guideline, you should minimize the scope of methods and classes whenever possible.

Recommendation - Model Objects are relatively simple "stand-alone" classes. They aren't coupled directly to either the user interface, or to the database. This makes them easy to unit test, using tools such as JUnit. It's highly recommended that you unit test your Model Objects. This will greatly increase your confidence in your validation code.

Simple Versus Complex Domain Models

• no use of the extends keyword (recommended, but not required)
• no hard-coding of 1..N relations (and similar) in domain classes

No extends Keyword

WEB4J isn't sensitive to the use of extends in your domain model. However, it's recommended that you avoid its use, since it's more difficult to work with. For further information, please see the topics in Joshua Bloch's Effective Java related to extends:

• Favor composition over inheritance
• Design and document for inheritance or else prohibit it
• Prefer interfaces to abstract classes

"There is no way to extend an instantiable class and add a value component while preserving the equals contract."

This means that the equals method is, in a sense, broken. If you need to extend a class, and add a new significant field, then it's not possible to provide a correct implementation of equals.

No 1..N Relations

Many are in the habit of hard-coding 1..N relations into the domain model, where one domain class has a List or Set of some other domain class. However, it can be strongly argued that it's better to avoid this, since it often doesn't correspond well with how different features use different data. For instance, take an example of a medical application, with 1 Doctor having N Patients. It's easy to imagine different features/screens having different needs:

• details for a single Doctor
• a listing of N Doctors
• a single Doctor with a listing of their N patients

Each feature uses different data. In some contexts, there's no need to refer to Patients at all. That is, there are common cases in which the underlying relation is of no relevance. To insist that the Doctor class must have an explicit, hard-coded relation to a Patient class disregards this fact. In effect, such a style promotes one particular use case to the detriment of the others.

On the other hand, if the Doctor and Patient classes are independent, then each feature/screen can simply retrieve the data as needed -- no more, and no less. If a feature needs to show a single Doctor along with a list of their Patients, then they are simply retrieved by the Action and placed into request scope. In effect, the relation is implemented in the Action, instead of the domain model. No wiring together of the objects is needed.

If you implement relations in your domain model, then you pay twice: first by defining an explicit link in your domain model, and then by ensuring that items are ignored in any context where they aren't needed. That's a lot of effort, for little benefit. In effect, this seems to create more problems than it solves.

There is a split among programmers between those who take data as king, and those who take the domain model as king. WEB4J agrees with those who take data is king, and as well with those who would like to keep domain classes as simple as possible.

Data Access Objects

Web4j has tools to help you implement persistence with a relational database, but you aren't forced to use them. If you want to use some other tool for persistence, then you can certainly do so. If you use web4j's data layer, then your DAO's will usually use the Db class.

Example code for some DAOs:

• VisitDAO
• RestoDAO
• RoleDAO

Recommendation - When using package-by-feature, your DAO classes should be package-private by default. You should increase their scope to public only when necessary. If a DAO method needs to be visible from another package, then you will need to change the scope of the class to public. However, this doesn't mean that all methods in the DAO should also be public. Rather, just increase the scope only for those methods that really need it.

Tip - If a package-private DAO is small, it's often perfectly reasonable to remove the DAO class entirely, and place data access methods directly in the Action class. This follows the general guideline that each of your classes should do a significant amount of work. This style is used in the Fish & Chips Club example application in several places (for example, the Rating feature).

The .sql files

Recommendation - Use one .sql file for each feature, in the same style as the example applications. Using a single .sql file for the whole application mixes together unrelated features. In addition, if more than one developer is working on the project, then developer contention for that single file will likely be a problem. It's often simplest to use a fixed, conventional name for each .sql file, such as statements.sql.

The problem arises of how exactly these SQL statements are referenced in your source code. Incorrect references in code should be detected as soon as possible. Detecting them at compile time would be best, and detecting them at runtime would be worst. The approach taken by WEB4J is to detect bad references at startup time. This is not quite as effective as detecting such errors at compile time, but it's much more effective than detecting them at runtime.

At startup, WEB4J will read in all .sql files located under /WEB-INF/. It will also scan your code for all public static final SqlId fields. Then, these two sets are compared. WEB4J strictly enforces a one-to-one relation between the SqlId fields defined in your code, and the named statement blocks defined in your .sql files.

Here's an example. The statement named ADD_COMMENT (appearing in an .sql file) must correspond exactly to a single SqlId object (appearing in your code). The text passed to that SqlId's constructor must exactly match the name of the SQL statement.

If any mismatch is found, then a RuntimeException is thrown, and your application will not be able to proceed. This protects you both from trivial naming errors and from 'cruft' (items that are no longer used), both in your code and in the .sql files.

The SqlId fields are what you use in code to reference items in your .sql files. They form the bridge between Java-land and SQL-land. For more information, please see the overview of the database package.

Tip - It's occasionally useful to disable an .sql file. To disable an .sql file, change its extension to .SQL (or anything else other than .sql). WEB4J will only read in files having the .sql extension.

Ordering Conventions

Column Order of SELECTs and Model Object Constructors - The columns in a SELECT ResultSet must match one-to-one with corresponding arguments of the target Model Object constructor. Using this convention means that you don't have to perform tedious mapping between columns and Model Object setters.

Example: note how the constructor of Resto has arguments that correspond one-to-one to the SQL statements named LIST_RESTOS and FETCH_RESTO.

Order of Parameters Passed From DAO - SQL statements usually have one or more parameters, denoted by '?' placeholders in the underlying SQL. When your DAO passes data to such SQL statements, their order is taken as matching one-to-one with the corresponding '?' placeholders.

Let's take an example. For this entry in an .sql file:

ADD_RESTO  {
 INSERT INTO Resto (Name, Location, Price, Comment) VALUES (?,?,?,?)
}

The corresponding DAO method is:

Id add(Resto aResto) throws DAOException, DuplicateException {
  Id result = Db.add(
    ADD_RESTO, 
    aResto.getName(), aResto.getLocation(), aResto.getPrice(), aResto.getComment()
  );
  return result;
}

Id add(Resto aResto) throws DAOException, DuplicateException {
  Id result = Db.add(
    ADD_RESTO, 
    aResto.getName(), aResto.getLocation(), aResto.getComment(), aResto.getPrice()
  );
  return result;
}

Warning - When testing, you should validate that the above conventions are being followed.
See below for more information.

Reports

Here are some examples:

• Report1Action
• Report3Action

If an appropriate Model Object already exists, then it should likely be used to return the required data, using the Db class.

If no Model Object exists for reporting a given set of data, then the Report class can be used. The Report class translates a ResultSet into a Map. It has several methods, corresponding to different policies for formatting the data.

Transactions

• TxSimple - for the simplest kinds of transactions, where no looping is involved. TxSimple uses ordering conventions to make your code as compact as possible. See UserDAO.delete(String) for an example.
• TxTemplate - an example of the Template Method design pattern. It defines all of the repeated code associated with transactions, while leaving you to implement a single abstract method, for performing the body of the transaction. When subclassing TxTemplate, you have the option of defining a private static nested class in your existing DAO, instead of creating a new top level class. This is simply because such classes are usually not very large. If you aren't comfortable with inner classes, you can certainly define an ordinary top-level class instead.
• TxTemplate will usually use DbTx. DbTx is almost exactly like Db. The difference is that Db uses connections created internally, while DbTx uses a shared connection passed by the caller.

Distributed Transactions

Connections and Multiple Databases

The database names defined by ConnectionSource are used in your application as prefixes to identify which database a given operation is intended for. These prefixes are used in two places: in .sql files and in the corresponding SqlId fields (see above).

An entry in an .sql file versus the default database looks like this (block name has no prefix):

ADD_COMMENT {
 INSERT INTO Discussion (Name, Body, CreationDate) VALUES (?,?,?)
}
...
public static final SqlId ADD_COMMENT =  new SqlId("ADD_COMMENT");

ACCESS_CONTROL.USER_LIST {
 SELECT Name, Password
 FROM Users
 ORDER BY Name
}
...
public static final SqlId USER_LIST =  new SqlId("ACCESS_CONTROL", "USER_LIST");

Conversions

Dynamic SQL

The most common occurrence of this is in search operations. Search operations are implemented with a SELECT, but they differ from other SELECTs in an important way. In many cases, the user input which defines the search criteria can come in a large number of combinations. In those cases, enumeration of all possible combinations in your .sql file is often impractical. The DynamicSql class was created to help with this issue. It allows you to programmatically add the WHERE and ORDER BY clauses to base SQL statements (defined in the usual way in an .sql file). In this case, the static parts of the statement are placed in the .sql file, and the dynamic parts are appended in code. Before you use the DynamicSql class, you must have a good understanding of SQL injection attacks.

To use DynamicSql correctly, you need to parameterize user input:

1. build a statement (partial or complete) having '?' placeholders for all user input
2. pass all the data input by the user as parameters to the above

See the example app's search feature for an illustration of using DynamicSql (especially the RestoSearchAction class).

There's another use case for DynamicSql as well. It's possible to create the entire SQL statement dynamically in code. In this case, there is still an entry in your .sql file, but it's an empty block. This allows you to to specify a non-default database name in the name of the block, if desired. It also allows WEB4J to apply the same processing as it usually does to your SqlId fields.

Database Configuration in web.xml

Postgres Driver - Internal Conversions

stringtype="unspecified"

Further Information

Request Processing

(Your code is almost always concerned with only the last 2 steps appearing above. You don't have to remember this structure. It's presented simply as background.)

By default, WEB4J uses a reasonable naming convention to map each incoming request to an Action. That convention is defined by RequestParserImpl. If you wish to use some other mechanism, then you will need to provide an alternate implementation of RequestParser.

Here's an example of the default Action mapping: given the Action class with fully qualified name of

hirondelle.fish.main.member.MemberEdit

/main/member/MemberEdit

This default mechanism also allows for simple explicit overrides: just add to your Action a String field of the conventional form:

public static final String EXPLICIT_URI_MAPPING = "/translate/basetext/BaseTextEdit";

Once the Action class corresponding to the underlying request is found, then WEB4J creates an object of that class, and calls its execute method. That method performs the desired operation, and returns a ResponsePage, which tells WEB4J how you want to render the final response. The ResponsePage points to a JSP, and controls the forward versus redirect behavior.

Application Security

It's strongly recommended that you review the various items below related to security.

WEB4J does its best to help you create secure web apps. In a few important cases it has built-in, default mechanisms which protect against certain kinds of attack:

• any incoming request URL that doesn't match the set of URLs expected by the application is automatically rejected.
• each action is mapped only to a specific, explicit set of request parameters. Any request parameter whose name doesn't match those on a white list, or whose value is not accepted by WEB4J's hard validation mechanism, is automatically rejected.
• WEB4J's data layer always uses PreparedStatement, never Statement. This helps protect your application from SQL injection attacks. See above for more information.
• the default implementation of ConvertParam defaults to not allowing String as a building block object. Using SafeText is encouraged instead. This protects against cross-site scripting attacks.

In other cases, however, only optional mechanisms are available. For example, using SafeText to model free-form user input instead of String is highly recommended, but not absolutely mandatory. If you use these optional mechanisms correctly, then they will increase your confidence in the security of your app. It's important to realize that using WEB4J doesn't guarantee that your app is secure.

(Indeed, it does not seem possible for a web app framework to address all security problems. For example, how can a framework prevent you from storing passwords in clear text, or from using http when https is needed?)

SQL Injection Attacks

For cases in which DynamicSql is used, please see the section on dynamic SQL for important information.

Cross-Site Scripting (XSS) Attacks

Cross-site scripting attacks are dangerous, common, and simple to perform. They are a major problem with web applications in general, and they should be taken very seriously. WEB4J takes the position that, in a web application, Strings containing free-form user input are a dangerous substance.

WEB4J provides SafeText as a replacement for String. SafeText will automatically escape special HTML characters in its toString() method. When needed, other escaping policies are available from SafeText, by calling its getXmlSafe() and getRawString() methods. The principal advantage is that JSP Expression Language can safely be used to render SafeText objects without worrying about special characters.

Tools such as <c:out> which manually escape free-form user input in the view will likely work - if you remember to use them. When using SafeText, however, the danger of forgetting to escape in the view does not exist. Hence, using SafeText will likely increase your confidence in the security of your application.

Warning - Many tools (including <c:out>) escape just 5 characters: &, <, >, " and '. However, it's not clear if escaping just those 5 characters is sufficient to protect you from XSS attacks. For example, the OWASP site lists 12 special characters that should be escaped. In the interest of providing maximum protection, WEB4J takes an even more aggressive approach, and escapes 33 special characters.

SafeText has a second line of defense as well. In addition to escaping special characters, it allows you to specify a "white list" of acceptable characters. The white list is defined by your implementation of the PermittedCharacters interface. The default implementation will often be satisfactory.

The ConvertParam interface defines how building block classes are created out of request parameters, before being passed to Model Object constructors. The default implementation always allows SafeText, but it only conditionally allows Strings. This is controlled by the AllowStringAsBuildingBlock setting in web.xml. By default, that setting is turned off. That is the recommended setting. It provides maximum protection against XSS attacks. If you wish to allow your Model Objects to be built with Strings, then you must explicitly allow it.

Recommendation - Using String as a building block class is discouraged. The SafeText class is recommended as a replacement for String, as a defence against Cross-Site Scripting (XSS) attacks.

Cross-Site Request Forgery (CSRF) Attacks

In WEB4J, you defend against CSRF attacks by performing the following:

• specify the correct POST/GET value for the method attribute of your forms. Any action which has a side effect should be a POST (edits to the database, logging off), while any other action should be be a GET (reports, listings, searches).
• always explicitly specify content-type, using an HTTP header. (For an example, see the Template.jsps in the example application.)
• configure a CsrfFilter in your web.xml. This filter requires two SqlId entries. (As with any SqlIds, you will need to provide the underlying SQL statements in an .sql file, and declare the SqlId constants in some class.)

Application Firewall

The default ApplicationFirewallImpl works closely with RequestParameter. If an Action uses request parameters (and they usually do), then it must state explicitly what request parameters it expects: what their names are and what basic sanity checks should be performed on their values. That is done by declaring RequestParameter fields in the Action of the form (example):

public static final RequestParameter COMMENT = RequestParameter.withLengthCheck("Comment");

Controlling Sessions

WEB4J helps in these ways:

• WEB4J itself will not create any sessions. It will only use sessions already created by the container during login.
• the optional SuppressUnwantedSessions filter disables URL rewriting entirely. URL rewriting is somewhat dangerous since it's a common vector used in session hijacking and session fixation. That is, the security risk from stolen session ids far outweighs other issues regarding session cookies. Of course, this means that the browser must have cookies enabled.

• when possible, the @page directive with session=false should be used in JSPs to minimize creation of unnecessary sessions.
• when the user logs out, your LogoffAction should explicitly destroy the session cookie (example).

In addition, the OWASP project recommends that a new session id should be created after every successful login. However, given the behavior of form-based login (as implemented in Tomcat, in any case) this doesn't seem possible to fully implement as desired. When Tomcat serves the login form, the session id is passed back to the client. Thus, the session id is served just before the user has successfully logged in, not after.

Logging User Data

WEB4J usually logs at the FINE level. Log entries containing user data, however, are always logged at the FINEST level. In production, where the logging level is almost always higher than FINEST, such data will not appear in the log files at all.

The following two items are strongly recommended:

• in production, the logging level should be set such that only a minimal amount of logging occurs.
• if your Actions log user data in any way, it should be logged at the FINEST level, to reduce the likelihood of sensitive data appearing in log files in production.

Preventing Spam

The SpamDetector interface and its SpamDetectorImpl help you prevent spam from getting into your database. Once you have decided on an implementation, it can be used in two ways.

• instruct the ApplicationFirewall to use the SpamDetector on every request parameter value. This is controlled by the SpamDetectionInFirewall setting in web.xml.
• for more finely tuned control, use Check.forSpam() to validate each parameter that may contain spam. Data that is modeled in a Model Object in a non-textual form, such as Integer or BigDecimal, does not need to be checked for spam in this way. This is because WEB4J internally converts such data from String into the target Integer or BigDecimal (or whatever), and spam will almost always cause such conversions to already fail.

Tip - Most spam contains links. If it's expected that no links will form part of any end user input, then it's best to use the first style described above, and let the ApplicationFirewall automatically check every request parameter value for spam. If there is a possibility of links appearing in user input, however, then the other style must be used.

Warning - The SpamDetector helps you prevent spam, but it's not guaranteed to completely prevent it. For public sites, you will often find that other tools (such as a CAPTCHA mechanism) will be required. WEB4J does not itself include a CAPTCHA mechanism, but its Predictions example app uses the recaptcha tool, (along with recaptcha4j by Soren Davidsen).

Fine-Grained Security Constraints

<!-- /webmaster/* only for 'webmaster' Role -->   
<web-app>
  <security-constraint>
   <web-resource-collection>
    <web-resource-name>Webmaster Module</web-resource-name>
    <url-pattern>/webmaster/*</url-pattern>
   </web-resource-collection>
   <auth-constraint>
    <role-name>webmaster</role-name>
   </auth-constraint>
  </security-constraint>
  ...
  <security-role>
   <role-name>webmaster</role-name>
  </security-role>
</web-app>

• noun - what is being acted on
• verb - what is being done to it; the operation, such as add, change, or delete.

An effective alternative is to use the <url-pattern> to represent both the nouns and the verbs:

• use a path in <url-pattern> to represent nouns (/blah/*)
• use an extension in <url-pattern> to represent verbs (*.add)
• use both a path and an extension to represent specific operations on specific items (../blah/SomeThing.delete)

When using this technique, incoming URLs appear as follows:

• ../MemberAction.list
• ../MemberAction.add
• ../MemberAction.fetchForChange?Id=3
• ../MemberAction.delete

The Member feature of the example app uses this technique.

When using this technique, you must also ensure that your <servlet-mapping> entries in web.xml allows for the various suffixes.

Restrict-by-User Constraints

While the Servlet Specification allows for security constraints based on role (restrict-by-role), it says nothing about security constraints based on user (restrict-by-user). Internal intranet applications usually have restrict-by-role constraints, but public web applications usually have restrict-by-user constraints.

There are a number of ways to implement restrict-by-user constraints.

Direct Links To User Table

Users <- PredictionList <- Prediction

Prediction.PredictionListFK => PredictionList.Id
PredictionList.UserFK => Users.Id

UPDATE PredictionList SET Title=? 
WHERE Id=? AND UserFK=?

DELETE FROM PredictionList 
WHERE Id=? AND UserFK=?

SELECT 
  Id, Title, CreationDate, UserFK
FROM 
  PredictionList
WHERE Id=? AND UserFK=?

This technique can't be used with INSERT operations, since they have no WHERE clause.

When a user logs in, the user login name (usually not the same as the id) is always placed into session scope by the servlet container. In most databases the user table will be normalized, such that the user id will be used as a foreign key in other tables (and not the user login name). When the WEB4J Controller detects successful logins, it calls your implementation of the LoginTasks interface. This is how you can add any 'extra' data to session scope - in this case, the user id. Thus, the user id can be easily passed to the above SQL statements.

It's important to realize that the user id is a server-side secret, and should never be displayed in a web page, or sent to the client in any way. It's critical for implementing restrict-by-user constraints, and the end user is not allowed to read or write its value. (Of course, the user's login name is not a server-side secret.)

Indirect Links To User Table

Using Subqueries

The most compact technique is to use subqueries; in particular, a 'scalar' subquery that returns a single value, the user's id or login name. If your database supports subqueries, then a WHERE clause can usually be added to link back explicitly to the owner. As in the simpler case of a direct link described above, both the operation and the restrict-by-user constraint can still be performed with a single SQL statement.

Here's a deletion operation that uses a subquery to ensure that the owner of the record is the same as a logged in user, named 'bob':

DELETE FROM Prediction 
WHERE Id = 65498
AND (
 SELECT 
  Users.LoginName 
 FROM 
  Users
 JOIN 
  PredictonList ON Users.Id = PredictionList.UserFK 
 JOIN 
  Prediction ON PredictionList.Id = Prediction.PredictionListFK
 WHERE 
  Prediction.Id = 65498
) = 'bob'

-- first define the subquery
PredictionOwner { 
 SELECT 
  Users.LoginName 
 FROM 
  Users
 JOIN 
  PredictonList ON Users.Id = PredictionList.UserFK 
 JOIN 
  Prediction ON PredictionList.Id = Prediction.PredictionListFK
 WHERE 
  Prediction.Id = ?
}

-- then reference the subquery in the top-level operation
DELETE FROM Prediction 
WHERE Id = ? 
AND (${PredictionOwner}) = ?

This technique can't be used with INSERT operations, since they have no WHERE clause.

Using web.xml and FetchIdentifierOwner

WEB4J offers a second way of implementing restrict-by-user constraints. This alternative can be used when:

• subqueries aren't supported by your database.
• you want to define in one place all URLs that must implement a restrict-by-user constraint, and simultaneously force corresponding Actions to validate the owner.

The general idea is to use validated proxies for the user id. Continuing the example used above, remember that the Prediction table does not link directly to the Users table, since it's 2 steps away, not 1:

Prediction.PredictionListFK => PredictionList.Id
PredictionList.UserFK => Users.Id

/PredictionAction.list?ParentId=5
/PredictionAction.fetchForChange?Id=7&ParentId=5
/PredictionAction.add?ParentId=5
/PredictionAction.change?Id=7&ParentId=5
/PredictionAction.delete?Id=7&ParentId=5

Validation can be done in 2 steps:

Step 1.
First, add an entry to web.xml of the form:

<init-param>
  <param-name>UntrustedProxyForUserId</param-name>
  <param-value>
   PredictionAction.*
  </param-value>
</init-param>

Step 2.
Second, the Action (in this example, PredictionAction) must now implement the FetchIdentifierOwner interface, which has a single method:

Id fetchOwner() throws AppException;

 SELECT 
   Users.LoginName
 FROM 
   PredictionList, Users
 WHERE
   PredictionList.UserFK = Users.Id AND 
   PredictionList.Id = ?

The Controller processes each request roughly as follows:

if the request uses an unvalidated proxy for the user id {
  cast the Action into FetchIdentifierOwner
  if the cast fails {
    fail: do not process the request
  else {
    get the result of fetchOwner()
    if it matches the user login name {
      success: continue processing the request
    }
    else {
      fail: do not process the request
    }
  } 
}

In summary, WEB4J defines these items for validating proxies for the user id:

• UntrustedProxyForUserId - an interface; defines which requests carry unvalidated proxies for the user id.
• UntrustedProxyForUserIdImpl - the default implementation, which uses the init-param in web.xml, as described above. (It's expected that the great majority of applications will use this default implementation.)
• FetchIdentifierOwner - implemented by Actions that need to validate a proxy for the user id.

Here's a comparison of the two techniques for implementing restrict-by-user constraints:

Regular DAOs

• each operation 'manually' performs the check on ownership
• a SELECT for the owner is first performed
• the result is compared to the id or login name of the current user
• if there is a match, then proceed with the core operation; otherwise it fails
• a transaction might be used to ensure a consistent view

Actions

Tip - It's useful to think of an Action as the public face of each feature. That is, someone reading the javadoc of the Action should have a clear idea of what the Action really does, and the important parts of its implementation. In WEB4J's example applications, the Actions usually declare the SqlId fields, and its javadoc has links to the underlying SQL statements and JSPs. This style allows most DAOs to remain package-private. (Unfortunately, it's not usually possible to make the Model Object package-private, since Model Object constructors and getXXX methods usually need to be public.)

• ActionImpl - many utility methods, and one abstract execute method. Forms the base class for the ActionTemplateXXX classes. If you need to define a new ActionTemplateXXX class, you should likely use ActionImpl as a base class as well.
• ActionTemplateListAndEdit - list, add, edit, and delete records.
• ActionTemplateSearch - suitable for search criteria forms that are used to perform a SELECT.
• ActionTemplateShowAndApply - show a form, possibly with data, and apply an update. An example use case is user preferences: from the point of view of each user, there is only one set of preferences. So, there is no need for a listing of numerous records.

Action methods tend to be 'branchy'. This is natural, since Actions need to handle all the different possible errors that can occur. Here are some example Actions:

• RestoAction
• CommentAction
• VisitAction

Convenient Object Key Names

• ITEMS_FOR_LISTING
• ITEM_FOR_EDIT
• DATA

Display Messages to the User

/** Update an existing {@link Visit}. */
protected void attemptChange() throws DAOException {
  boolean success = fVisitDAO.change(fVisit);
  if (success){
    addMessage("Visit to _1_ changed successfully.", fVisit.getRestaurant());
  }
  else {
    addError("No update occurred. Visit likely deleted by another user.");
  }
}

The example application displays messages with a reusable .tag file (see /WEB-INF/tags/displayMessages.tag), always placed at the top of a templated page. The alternative style of placing error messages near form controls is sometimes problematic:

• in the case of longer forms that require scrolling, there's no way to ensure that the user actually sees a message placed only beside a control. Thus, it seems best to at least always include some kind of message at the top of the page, in addition to any message appearing next to the control.
• form layout can easily degrade when large message text is inserted into the flow of a form layout.

Building Model Objects

Example

public static final RequestParameter MEMBER_ID =  RequestParameter.withLengthCheck("Id");
public static final RequestParameter IS_ACTIVE = RequestParameter.withLengthCheck("Is Active");
public static final RequestParameter NAME = RequestParameter.withLengthCheck("Name");
...
/** Ensure user input can build a {@link Member}.  */
protected void validateUserInput() {
  try {
    ModelFromRequest builder = new ModelFromRequest(getRequestParser());
    fMember = builder.build(Member.class, MEMBER_ID, NAME, IS_ACTIVE);
  }
  catch (ModelCtorException ex){
    addError(ex);
  }    
}
...
private Member fMember;

public Member (Id aId, String aMemberName, Boolean aIsActive) ... {
  ...
}

There are numerous examples of building Model Objects in this way in the example application:

• RestoAction
• RatingAction
• RestoSearchAction

Populating Forms

There are two use cases for the Populate tag: when there's no editing of an existing record, as in

<w:populate>
  ..form body...
</w:populate>

<w:populate using='some identifier'>
  ..form body...
</w:populate>

<w:populate using='item'>
   ...
   <input name="Message" type="text">
   ....
</w:populate>

Forms that edit the database must have method=POST, while forms that perform searches or implement reports must have method=GET.

Configuring web.xml

Tip - When getting started, the following items are the most important: MailServerConfig, MailServerCredentials, Webmaster, ImplicitMappingRemoveBasePackage, LoggingDirectory, TroubleTicketMailingList, DefaultUserTimeZone, and RedirectWelcomeFile. The rest of the items can usually be ignored.

If you're running in a Servlet 3.0 environment, and you find it annoying that these framework settings get in the way of your own config in web.xml, then you have the option of re-packaging these settings into a META-INF/web-fragment.xml file, and embedding it yourself into web4j.jar. (See § 8.2 of the Servlet 3.0 spec for more information.)

Items Having No Default Value

Webmaster
Email address of the application admin. No default value for this item. If TroubleTickets are emailed by WEB4J, they will be sent to this address. If emails are sent by WEB4J to other parties, then they will have this as the "from" address. This email address must be in the same domain as the MailServerConfig below.

ImplicitMappingRemoveBasePackage
Example value: 'com.blah' - do not include any trailing dot. No default value for this item. By default, the RequestParserImpl class will map Action classes implicitly to a specific URI. This specific URI is constructed by taking the Action class name, and removing the 'base' package, defined by this setting.

Example: given the settings in the example application, the class:

hirondelle.fish.main.home.HomePageAction

/main/home/HomePageAction

Implicit mapping can always be overridden by specifying in the Action a conventional field named EXPLICIT_URI_MAPPING, of the form:

public static final String EXPLICIT_URI_MAPPING = "/main/Blah";

Items Having A Default Value

MailServerConfig
Controls how web4j will send emails on the application's behalf. Default value: NONE. Used to send trouble-ticket emails. Name-value pairs of N properties to be used when connecting to the outgoing mail server. Each name-value pair must appear on a separate line. Example:

            mail.host = smtp1.blah.net
            mail.smtp.port = 465

MailServerCredentials
The user name and password for the outgoing mail server, separated by a pipe character '|'. Default value: NONE. If the special value "NONE" is supplied, then no credentials will be passed to the mail server (which, in some cases, can still work).

LoggingDirectory
Location for logging file. Default value: NONE. Example value: C:\log\fish\. Used by LoggingConfigImpl. Must end with a directory separator. Special value of 'NONE' will disable LoggingConfigImpl, such that it will not perform any logging config in code.

LoggingLevels
Logging levels for various loggers. Used by LoggingConfigImpl. Comma-separated list of items, in the same format as JDK logging.properties files:

com.wildebeest.myapp.level=INFO, hirondelle.web4j.level=FINE

hirondelle.web4j.level=CONFIG

TroubleTicketMailingList
Comma-delimited list of one or more email addresses to be notified when a problem occurs. If TroubleTickets are emailed, they will be sent to these addresses. Default value 'NONE'.

MinimumIntervalBetweenTroubleTickets
Time interval in minutes. Suggested value: 30. Default value: 30.
TroubleTickets are expected to be rare. It's possible, however, that TroubleTickets could be generated in large numbers in a short period of time. For example, if the database connection is lost, then a trouble ticket is generated with almost every request. This parameter is used to throttle down the sending of such emails.

PoorPerformanceThreshold
Time interval in seconds. Suggested value: 20. Default value: 20.
If any request processed by the Controller takes more than this number of seconds, then an email will be sent to the configured Webmaster. The email will state the number of milliseconds taken to process the request. Often, poor performance is caused by degradation not in the web application itself, but rather in the environment in which it's running.

MaxHttpRequestSize
Suggested value: 51200. Minimum value: 1000. Default value: 51200.
Maximum size in bytes of HTTP requests carrying no uploaded files. If such a request has a total size in excess of this value, then it will be discarded by WEB4J, and will not be made available to the application. This is intended only for requests whose unusually large size identify them clearly as hack requests.

MaxFileUploadRequestSize
Suggested value: 1048576. Minimum value: 1000. Default value: 51200.
Maximum size in bytes of HTTP requests carrying uploaded files. If such a request has a total size in excess of this value, then it will be discarded by WEB4J, and will not be made available to the application. This is intended only for requests whose unusually large size identify them clearly as hack requests.

MaxRequestParamValueSize
Suggested value: 51200. Minimum value: 1000. Default: 51200.
Maximum size in bytes of HTTP request parameter values. This maximum is applied as a hard validation by ApplicationFirewallImpl, and only against items created using the RequestParameter.withLengthCheck(String) factory method. This is meant as a kind of default sanity check on parameter values, when no other (more stringent) means of performing hard validation is suitable.

SpamDetectionInFirewall
Permitted values: ON and OFF. Default value: OFF. When ON, then the default implementation of ApplicationFirewall will use the configured SpamDetector to check the values of all request parameters. If spam is detected, then an unpolished response will result.

FullyValidateFileUploads
Permitted values: ON and OFF. Default value: OFF. Controls the behavior of ApplicationFirewallImpl for file upload requests. When ON, then ApplicationFirewallImpl will treat file upload requests the same as any other request, and will check request parameter values in the usual way. When OFF, then it will not perform such validation for file upload requests.

The problem addressed here is that file upload requests are fundamentally different from regular requests. In particular, access to request parameters is possible only if the file upload request has been 'wrapped' in some way, such that request parameters can be read in the usual way. This setting should be ON only if such request wrapping has been performed. Otherwise, it should be OFF.

AllowStringAsBuildingBlock
Permitted values: YES and NO. Default value: NO. When YES, then the default implementation of ConvertParam will allow String as a building block object, along with Integer, Date, and so on. When set to NO, you will not be able to pass a String to any Model Object constructor, and you will need to use SafeText instead. The idea here is that using SafeText is preferable to String. Since String does not escape special characters, it's vulnerable to Cross-Site Scripting attacks.

UntrustedProxyForUserId
Default value: empty String. Lists all URLs having a Data Ownership Constraint implemented with an untrusted identifier. The syntax of this entry is described by UntrustedProxyForUserIdImpl.

CharacterEncoding
Character encoding used in the application. Default value is 'UTF-8'. This setting MUST match the character encoding used by the database. (Databases often use the term 'character set' for character encoding.) Character encodings define how bytes are translated into text. For more information, see the this link.

This setting will be used by the Controller to process requests. In addition, it will be used to create an application scope attribute named web4j_key_for_character_encoding. It's highly recommended that the value of this attribute be placed in template JSPs, so that all served pages will explicitly state the desired character encoding to be used by the browser. (This also increases security.)

DefaultLocale
Default Locale used by the application. Default value: en. This setting allows independence of the server JRE's default Locale, and of HTTP headers. See LocaleSource for information on how a Locale is derived from an underlying request. Example values : en, fr. See also: java.util.Locale for possible values of Locale identifiers, and Util.buildLocale(String).

DefaultUserTimeZone
Default TimeZone used by the application. Default value: GMT. This setting allows independence of the server JRE's default TimeZone, and of HTTP headers. See TimeZoneSource for information on how a TimeZone is derived from an underlying request. Example values: 'Canada/Montreal', 'UTC'. See also: java.util.TimeZone for possible values of TimeZone identifiers, and LocaleSource.

TimeZoneHint
Example values: 'NONE', 'Canada/Montreal', 'UTC'. Default value: 'NONE'. See java.util.TimeZone for possible values of TimeZone identifiers. This item is passed as hint to the database driver, for all java.util.Date columns encountered by the WEB4J data layer. Overrides the JRE default time zone. See: PreparedStatement.setTimestamp(int, Timestamp, Calendar) and ResultSet.getTimestamp(int, Calendar). The special value 'NONE' is used when the default JRE time zone suffices, or when the database already stores time zone information in the desired manner.

DecimalSeparator
Comma-separated list of names of decimal separator characters to be accepted during user input of numbers having a fractional part. Permitted values : 'PERIOD', 'COMMA', 'PERIOD,COMMA'. Default value: 'PERIOD'. Used for conversions to BigDecimal and Decimal values. This item is not affected by Locale.

BigDecimalDisplayFormat
Format for display of BigDecimal and Decimal values in reports. Default value: #,##0.00. See java.text.DecimalFormat for possible values. This pattern is non-localized. It's combined with a Locale to produce a locale-sensitive format.

BooleanTrueDisplayFormat
The text for presenting boolean 'true' values in a report. Example values: Yes, True, Oui.

The HMTL spec states that form controls can appear outside of a form. One may specify a checkbox, as in:

<![CDATA[
  <input type='checkbox' name='true' value='true' checked readonly notab>
]]>

The above is in fact the default value for this item. (CDATA is needed here because of the presence of special HTML characters.) Note that JSTL's c:out tag will need escapeXml set to false, in order to render such items as desired.

Another option is to use an IMG tag containing the desired image.

BooleanFalseDisplayFormat
The text for presenting boolean 'false' values in a report. Example values: No, False, Non.

See BooleanTrueDisplayFormat for more information on this style.

The default value for this item is:

<![CDATA[
  <input type='checkbox' name='false' value='false' checked readonly notab>
]]>

EmptyOrNullDisplayFormat
The text to display when an item has no value. Default value: '-' (a single hyphen). Many browsers do not render empty TABLE cells very well. The CSS 'empty-cells' property is the recommended way of controlling this behavior. As an alternative, this setting may be used to specify text representing empty or null items in a report. Example values: '-', '...'

IntegerDisplayFormat
The format for presenting integer quantities. Default value: #,###. See java.text.DecimalFormat for possible values. This pattern is non-localized. It's combined with a Locale to produce a locale-sensitive format.

IgnorableParamValue
Any parameter which takes this value will be ignored, by having its value replaced by an empty string, in RequestParser. Default value is an empty String.

Intended solely for SELECT tags, which show a list of items. Even when no selection is made by the user, most browsers will simply pass the first item in the SELECT as the parameter value. If such behavior is undesired, then specify this parameter, and put the IgnorableParamValue as the first item in the SELECT tag.

Typically, one would place unusual characters in IgnorableParamValue, to greatly reduce the likelihood of conflict with regular user input. You may specify an empty value as well, which corresponds to a blank initial value in the list.

In a multilingual application, this setting must not contain translatable text. For example, '----' or an empty string would be an appropriate value for a multilingual application.

IsSQLPrecompilationAttempted
Value: (true | false). Default value: true. If true, then upon startup WEB4J will attempt to precompile SQL statements appearing in the .sql file(s), by calling Connection.prepareStatement(String), and detecting if an SQLException is thrown. Any detected errors are logged as SEVERE.

It's important to note that some drivers/databases will NOT throw an SQLException even when the text is syntactically invalid. (This can be determined with a simple test.) Hence, this is always an "attempt" to precompile.

The idea here is to be defensive, and to attempt to detect errors as soon as possible. In addition, this service is very useful when porting an application to a new database: one may start with the "old" statements, and begin by seeing if they compile versus the "new" database. Recommended value is 'true', unless certain the driver/database does not support precompilation in this manner. See SqlId for more information.

MaxRows
Upper limit on the number of rows to be returned by any SELECT statement. Default value: 300. Serves as a defensive safety measure, to help avoid operations which may take an excessively long time. Set value to '0' to turn off this limit. This value is passed to Statement.setMaxRows(int).

FetchSize
Hint sent to JDBC driver regarding the number of rows to be returned when more rows are needed. Default size: 25. Applies only to SELECT statements. Set value to '0' to turn off this hint. This value is passed to Statement.setFetchSize(int).

HasAutoGeneratedKeys
Value: (true | false), ignores case. Default value: false. Indicates if the database and driver support the method Connection.prepareStatement(SqlText, Statement.RETURN_GENERATED_KEYS).

ErrorCodeForDuplicateKey
The error code(s) returned by SQLException.getErrorCode() when trying to add a duplicate record. Default value: 1 (Oracle's value). If you wish to treat more than one error code as being related to duplicate key constraints, you may specify a comma-delimited list of integers. If this error code is unknown for a given database, you may determine its value by exercising your application, and deliberately causing the error to happen. When web4j detects these error codes, then it will throw a DuplicateException, instead of a DAOException.

Some confirmed values for this item:

Some unconfirmed values for this item:

ErrorCodeForForeignKey
The integer error code(s) returned by SQLException.getErrorCode() when a foreign key constraint fails. If you wish to treat more than one error code as being related to foreign key constraints, you may specify a comma-delimited list of integers. If this error code is unknown for a given database, you may determine its value by exercising the example application, and deliberately causing the error to happen. When web4j detects these error codes, then it will throw a ForeignKeyException, instead of a DAOException.

Some example values (please confirm using your database documentation):

SqlFetcherDefaultTxIsolationLevel
The default transaction isolation level used for queries. Default value: DATABASE_DEFAULT. The permitted values are:

For more information, see TxIsolationLevel, and java.sql.Connection. It's important to confirm support for these levels with your database documentation.

SqlEditorDefaultTxIsolationLevel
The default transaction isolation level used for edit operations. Default value: DATABASE_DEFAULT. The permitted values are:

For more information, see TxIsolationLevel, and java.sql.Connection. It's important to confirm support for these levels with your database documentation.

DateTimeFormatForPassingParamsToDb
Formats used when passing DateTime objects as parameters to SQL statements.
Default value: YYYY-MM-DD^hh:mm:ss^YYYY-MM-DD hh:mm:ss

Three formats must be specified here, corresponding to 3 common cases (in this order):

• date only (year-month-day are present, but no other units)
• time only (hour-minute-second are present, but no other units)
• both date and time (all units are present)

The format Strings are those defined by DateTime. The 3 format Strings are separated by a ^ character.

If a DateTime parameter to an SQL statement needs to be passed in a format which is not specified here, simply format the DateTime in your code as desired, and pass the result to the database as a fully formatted String, instead of a DateTime.

RedirectWelcomeFile
Simple servlet for redirecting directory requests to the home page Action. The source code underlying this item is provided with the example application. It doesn't form part of the WEB4J core. Example value appropriate for development:

http://localhost:8080/fish/main/home/HomePageAction.show

Configuring Database Items in web.xml

100~Translation=200~AccessControl=300

There is a single exception - the TimeZoneHint setting can only take a single value, to be applied across all databases. (The reason for this unfortunate exception is simply that changing it would have a large number of ripple effects, and would represent too much pain for too little gain.)

This syntax is particularly useful for applications which use not only multiple databases, but also multiple database vendors (MySQL and PostgreSQL in the same application, for example).

Configuring Implementation Classes

The BuildImpl class controls the lookup of implementations. BuildImpl makes a distinction between three kinds of implementations, described below.

Default Implementations

Conventional-Name Implementations

The conventional package name is always 'hirondelle.web4j.config', while the conventional class name varies. Please see below for a complete listing.

Arbitrary-Name Implementations

<init-param>
  <param-name>ImplementationFor.hirondelle.web4j.ApplicationInfo</param-name>
  <param-value>com.xyz.MyAppInfo</param-value>
  <description>
    Package-qualified name of class describing simple, 
    high level information about this application. 
  </description>
</init-param> 

Upon startup, the BuildImpl class searches for implementations in the following order:

1. Arbitrary-Name implementation
2. Conventional-Name implementation
3. Default implementation (if it exists)

See BuildImpl for further information.

Recommendation - The conventional-name style is preferred over the arbitrary-name style. The conventional-name style takes less effort, and does not require an associated entry in a text file.

Listing of Interfaces and Conventional Names

Multilingual Apps

WEB4J's translation package has a number of items to help you implement multilingual applications. The most important item is the Translator interface. This interface will usually be implemented using either ResourceBundles or a database. In addition, this interface works closely with your implementation of the LocaleSource interface, which defines the end user's preferred language.

Recommendation - Using a database to store translations has numerous advantages over using ResourceBundle, since ResourceBundle has a number of annoying defects. For illustration of using a database, please see the Fish & Chips Club example application. (Please note that in the Fish & Chips Club, not all French translations are actually provided. This is to allow you to exercise the mechanism for adding translations at runtime.)

Base Text

• ordinary, natural language (such as 'Add/Edit').
• coder keys (such as add.edit.button).

Natural-language base text and coder-key base text differ slightly in their behavior. To illustrate this difference, let's take an example of an app that is developed in English and presented to the user in two languages, English and French. In this case, English is the base language. When the user's language preference is English, any natural language base text such as 'Add/Edit' can be presented to the end user as is, without further processing. However, if a coder key such as 'add.edit.button' is used instead, then such text is always going to need further processing, even when the user's preferred language is English. One might think of coder keys as being in a kind of special language known only to the programmer. Thus, coder keys always need translation, even into the base language of the application.

Recommendation - Natural language base text should be preferred over coder keys. When reading source code, natural language base text is more legible than coder keys. In addition, coder keys cannot be presented as-is to the end user, so they increase the effort needed to implement translation.

Minimally Invasive

The following tags implement translation in your JSPs:

• Text - translate a single item.
• TextFlow - translate all free flow text in entire sections of static markup.
• Tooltips - translate all TITLE, ALT, and submit-button VALUE attributes appearing in entire sections of markup.
• Messages - translate and present the items in a MessageList, containing error and information messages.

Using Your App to Assist in Translation

Storing Translations in Application Scope

Storing translations in application scope allows you to avoid the cost of repeated database lookups for data which rarely changes.

Using One JSP per Language

Such a design is implemented by subclassing the Controller. By providing an override of its swapResponsePage method, you can alter the default mechanism used by WEB4J to find the ResponsePage returned by your Actions. For example, a page referenced as 'Blah.jsp' in your Action can be changed by your custom Controller, late in processing, to actually refer to some other item such as 'Blah_en.jsp' or 'Blah_fr.jsp', according to the given Locale. (See LocaleSource for more information on Locales.)

The Visit feature of the Fish & Chips Club example application can be used to demonstrate this technique. To exercise it, you simply need to change the servlet-class entry in web.xml, to point to the Visit feature's CustomController.

Encoding Issues

When the same encoding is not used with 100% consistency across all parts of your application, then your users will see annoying '?' (or similar) symbols where regular text should appear. Such problems are common, and they're very annoying.

The primary defense here is to use the UTF-8 encoding. UTF-8 can represent almost any kind of text. In a perfect world, all tools would default to UTF-8. In practice, the problem is that many tools default to non-UTF-8 encodings. For example:

• the Servlet Specification itself has ISO-8859-1 as its default encoding
• the Windows operating system has CP1252 as the default encoding

• HTTP request headers and response headers can include an accept-charset header
• databases can use an encoding scoped to various levels: server instance, database, table, column
• database drivers can use an encoding when connecting to the server
• the first line of an XML file can state an encoding
• every JRE has a default encoding (stored in the System property named file.encoding)
• HTML <form> tags with an accept-charset attribute specify an encoding (to override the default)
• you can save your JSP files during development with a specific encoding. You will often need to communicate that encoding to the container using various means (see below).

Page Character Encoding

One of the problems with encodings is that they're hard to determine precisely simply by examining the raw bytes. Hence, various means have been concocted to get around the fact that encodings are not stated explicitly within the bytes themselves. For example, the first line of an XML file can appear as:

<?xml version="1.0" encoding="UTF-8"?>

It's natural to regard these encoding settings as specifying the encoding. But that's false; it's not a specification of the encoding, it's a (possibly incorrect) attempt to communicate the encoding. There's a big difference. (In practice, there are many such instances of mismatch between the stated and actual encoding.)

The JSP specification allows you to communicate the page character encoding in various ways. For JSP files that use standard (non-XML) syntax, the logic used to determine the intended page character encoding has the following order of evaluation:

1. JSP config in web.xml:
  <jsp-config>
    <jsp-property-group>
     <url-pattern>/*</url-pattern>    
     <page-encoding>UTF-8</page-encoding>
    </jsp-property-group>
  </jsp-config>
2. <%@ page pageEncoding="UTF-8"%> 
3. <%@ page contentType="text/html;charset=UTF-8"%> 
4. Default to ISO-8859-1

1. <%@ tag pageEncoding="UTF-8"%>
2. Default to ISO-8859-1

Serving JSON, XML, and so on

The only job of the JSP, in this case, is to let you control header information. Even though you are serving structured data, that data is still being passed around using HTTP, and every HTTP response should include headers for the content-type and encoding.

The Predictions example app has a simple example of serving JSON data. It's implemented by:

1. a JSP file named /pub/view.json in the app's root directory
2. a servlet-mapping entry in web.xml, which tells the container that .json files are to be treated as a JSP
3. an action class in the hirondelle.predict.pub.json package

Of course, the exact same technique can be used to serve data in any other format.

Logging

The WEB4J framework classes (and the example apps) use JDK logging. It's recommended that you configure JDK logging to view WEB4J's logging output, by configuring the JDK logging tool. You might get away with configuring logging.properties, but in a shared environment on a server, that will likely be inadequate: the Handlers (output files) defined in logging.properties are per JRE, not per web application. That means that different web apps will output to the same logging file.

To address that problem, the LoggingConfig interface and its default LoggingConfigImpl have been provided to allow a programmatic way of configuring logging (if you will permit that somewhat contradictory expression). This programmatic style is per class loader, not per JRE, so its Handlers (output files) are not shared between applications.

The default LoggingConfigImpl has the interesting property of allowing you to log records with a time stamp derived from your application's fake system clock. This is because it uses a TimeSource.

Start-up Tasks and Code Tables

The most common start-up task involves code tables. A code table is an enumeration or list of related items in the problem domain. For example, these kinds of items might be implemented as code tables:

• geographical divisions (countries, provinces, states, and the like)
• types of payment (credit card, cash, and so on)
• the names of months (January..December)

The Code class is provided to help you implement code tables. You are not obliged to use it.

In a multilingual application backed by a database, another common startup task is to place all translations into application scope. Thus, your Translator implementation can access in-memory data, instead of continually querying the database for these relatively unimportant items which rarely change.

The single method of the StartupTasks interface has two arguments. The second argument is the name of a database, as you have defined in your implementation of the ConnectionSource interface. Upon startup, WEB4J attempts to connect to each of your databases. As each attempt succeeds, the framework will pass the database name to your StartupTasks implementation.

Your implementation should follow this form:

public void startApplication(ServletConfig aConfig, String aDbName) throws DAOException {
  if (!Util.textHasContent(aDbName)){
    //tasks not related to any db - this is called first
  }
  else if (ConnectionSrc.DEFAULT.equals(aDbName)){
    //tasks related to this db
  }
  else if (ConnectionSrc.TRANSLATION.equals(aDbName)){
    //tasks related to this db
  }
}

What happens if one or more databases are down when your app starts? If that occurs, the framework will try to connect to any remaining 'bad' databases, with each incoming request. If a connection is found, then, as usual, the framework will pass the database name to your StartupTasks implementation.

File Uploads

Version 3.0 of the Servlet API (published January, 2010) has good support for file uploads. If it's available to you, you should use it to access uploaded binary data.

Servlet API version 2.5 (upon which web4j is still based) has surprisingly poor support for file uploads. When using Servlet 2.5, third-party tools such as the Apache Commons FileUpload tool are needed. This technique uses a filter and a request wrapper, and is demonstrated in the file upload feature of the Fish & Chips Club example app.

Regardless of which technique you use to access the uploaded data, it's made available to your Action as an InputStream. Web4j includes InputStream as one of its base building block classes, for constructing Model Objects, and for passing data into and out of the database.

When an InputStream is passed to the database, setBinaryStream(int, InputStream) is used. This method is available in Java 6+, and may not be supported by older drivers. For example, Tomcat 6 has an older connection pool implementation, and it doesn't support the above method, while Tomcat 7 has no such problem. Note that this issue is related to the level of support of JDBC, not of the Servlet API.

Form controls: browsers don't seem to respect the value attribute of form input controls. This makes it impossible for web4j (or any other tool) to dynamically pre-populate the selected file. Thus, any update operations on binary data will usually need some extra care. (The simplest implementations would provide only add and delete operations, with no update operation as such.)

The RequestParser class is able to translate RequestParameter objects into building block classes. The one exception is InputStream. When you need to build a Model Object that takes an InputStream, you will have to pass the InputStream directly, without going through an intermediate RequestParameter. This minor nuisance will be removed when web4j moves to Servlet 3.0 as its base API.

Paging

The Pager tag lets you emit links corresponding to the first, next, or previous page. The links don't perform the actual 'subsetting' of the data. That's done elsewhere, in one of 3 possible places:

• in the JSP itself, using <c:out>
• in a DAO method
• in a SELECT statement

Please see the Pager javadoc for further information. As well, the Fish & Chips Club example application implements paging in its Discussion feature, where users post comments to a simple message board.

Warning - The paging mechanism is a bit confusing. It depends on several items cooperating together: request parameters, the Pager tag, and the chosen subsetting mechanism. The subsetting mechanism in turn comes in 3 different styles, as stated above. Please read the Pager javadoc carefully to make sure you understand how it works.

Miscellaneous Tags

• AlternatingRow - provides styling of alternating rows in a table listing, to increase legibility. The example app's Resto feature uses this tag in its listing.
• HighlightCurrentPage - alters links to indicate the current page. In the example app, the Template.jsp files use this tag to implement menus.
• ShowDate - renders dates in various ways. In the example app, the Visit feature uses this tag to render dates in a particular format.
• ShowForRole - shows or hides its body according to the roles assigned to the logged in user, or simply whether or not they are logged in.
• TagHelper - not a tag, but a base class for creating tags

Double-Submit Issue

In the following cases, such double-submits are not an issue:

• forms with method='GET', which should only represent actions which can safely be repeated.
• forms for delete operations can usually be safely repeated; the second delete is merely a no-operation.
• forms for update operations can also be safely repeated; the second update merely does the same thing as the first, and will have no ill effect on the final state of the data.

• if possible, ensure that uniqueness constraints on the business data are applied, usually with a UNIQ constraint in the database. If a double-add happens, an exception regarding the violated constraint will be thrown; if that exception occurs, then the app can simply skip the add operation (since it has already been performed), and proceed as usual. It's not even necessary to inform the user that this has occurred.
• use javascript on the client to disable the submit button immediately after it is clicked
• serve each form-for-add with a new nonce token, an 'add-token', and record when each add-token is consumed by a successful add operation. Each add operation must first confirm that its add-token has not already been consumed; if it has been consumed, then a double-add has been detected, and the (second) add operation should simply be skipped. (Note that the token used here is distinct from the nonce token used by the CSRF filter, and serves a different purpose.)

Using the Database Layer Outside of a Servlet

When you use the database layer outside of a servlet context, you need to supply an implementation of ConnectionSource (as usual), and two collections. The collections define two things:

• a Map of configuration settings as name-value pairs (corresponding to settings supplied in web.xml)
• a List of Strings containing SQL statements (each String corresponding to the contents of an .sql file)

Here's an example:

//config settings 
Map<String, String> settings = new LinkedHashMap<String, String>();
//theses settings override the default settings
settings.put("Webmaster", "blah");
settings.put("ImplicitMappingRemoveBasePackage", "blah");
//no real need to use SafeText for a command line app
settings.put("AllowStringAsBuildingBlock", "true"); 
//optional logging settings
settings.put("LoggingDirectory", "C:\\log\\blah\\"); 
settings.put("LoggingLevels", "hirondelle.web4j.level=FINEST");
settings.put("HasAutoGeneratedKeys", "true");
//override any other default settings here

//fetch the raw SQL statement data 
//the FindSql class is defined by your app; it scans 
//the local file system for .sql files
FindSql findSql = new FindSql(new File(System.getProperty("user.dir") + "\\bin\\"));
List<String> rawSqls = findSql.scan();

//pass the config data and sql statements to this single call
Db.initStandalone(settings, rawSqls);

//now you can use the database layer in the usual way

ImplementationFor.hirondelle.web4j.database.ConnectionSource

If you use the Reports class, then you may also need to supply an implementation for a second interface: DateConverter.

In the above example, a FindSql class is referenced. You will need to create such a class, to find the SQL statements defined by your app. Typically, that class will need to scan the file system under a specific root directory (or perhaps in a jar), find all the .sql files, and read in their content using the correct encoding. Each file will be represented as a single String in the list passed to the Db class. In the context of a web app, this is performed by the framework upon startup. The reason this is possible for a web app is that the file and directory structure is well-defined for web apps, and the framework knows where to look for things. Outside of the servlet context, however, the location of that data becomes ambiguous, so your app needs to supply it explicitly.

Using the above code, the initialization tasks performed by Db.initStandalone are not as complete as those performed in a servlet context. In short, the validation and logging tasks are skipped. (The reason is the same as above: the location of items is ambiguous). For example, the SqlId's used in your code are not matched one-to-one with the corresponding names in your .sql files. In the absence of such validations, any errors, if present, will appear at run-time instead of at start-up.

Utilities

• Args - checks parameters in a compact style.
• Consts - commonly used constants. Items in this class are suitable for static imports into your classes.
• EscapeChars - escapes special characters in various ways.
• Stopwatch - times the execution of sections of code to microseconds.
• Util - large number of utility methods.
• WebUtil - utility methods specific to the context of the web.

Unit Testing

Different organizations take different approaches to unit testing. Some aggressively unit test all classes, while others prefer to focus on areas most likely to contain bugs. (Many casual applications don't include any unit tests at all.)

For an illustration of unit testing, please see the following packages in the Fish & Chips Club example application:

• hirondelle.fish.test
• hirondelle.fish.test.doubles
• hirondelle.fish.main.Member

To run the tests in the Fish example app, perform the following:

• edit TESTAll.setRootDirectory() to use the root directory of the web app
• edit FakeConnectionSrc to add your credentials for database access
• ensure the database driver is on the classpath
• run TESTAll.main()

WEB4J helps you implement unit tests by providing test doubles. Using the terminology of Gerard Meszaros and Martin Fowler, these test doubles are fake objects (not mock objects). Such fake objects serve to mimic the real runtime environment of a class, using implementations you can control as desired.

The example application includes source code for the following fake objects:

• FakeServletConfig
• FakeServletContext
• FakeRequest
• FakeResponse
• FakeSession

These fake objects make a number of simplifying assumptions. The most important is that only a single thread of execution is assumed.

Also included is the source code for some utilities you may find useful:

• FakeConnectionSrc - a fake implementation of ConnectionSource, which allows running tests without using JNDI or an externally defined connection pool.
• FakeDAOBehavior - controls how fake DAOs behave, allowing you to mimic various kinds of exceptions. This class stores data as simple System properties.

Where To Place Unit Tests

Initializing WEB4J Classes

See the TESTAll.initControllerIfNeeded() method in the example app for an illustration.

There are also 2 methods in BuildImpl which were created specifically for creating a proper test environment:

• adHocImplementationAdd(Class, Class)
• adHocImplementationRemove(Class)

Unit Testing Model Objects

Unit Testing DAOs

• order your tests such that any records added during tests are simply deleted at the end
• manually run a database script to return the database to a fixed initial state
• perform the unit tests inside a transaction, and rollback the transaction at the end of the test

DAOs and Ordering Convention Errors

It's useful to test explicitly for ordering errors. In general, you will find more ordering errors if you test with data that clearly distinguishes between different fields/columns.

To illustrate, here's an example using two fields/columns in Member objects (taken from the example app):

If there is a permutation error involving Disposition and IsActive, and if you use similar values for these fields during testing, then the permutation error will not be detected. If you use distinct values, then any permutation error will be detected.

So, the guideline is to make sure your test data distinguishes clearly between fields/columns. It's likely best to avoid common, generic values such as 0, 1, nulls, and empty Strings.

Fake DAOs

• Action unit tests are easily repeated, since they have no persistent side effects
• Action unit tests might execute significantly more quickly
• fake DAOs can be made to explicitly throw specific exceptions, perhaps more easily than the real DAO

Instead of fake DAOs, you might also consider a fake relational database.

Unit Testing Actions

• they need a fake request and a fake response
• it might be best for them to use fake DAOs as well (see above)

Fake System Clock

• System.currentTimeMillis()
• the default constructor for the Date class (which in turn uses System.currentTimeMillis())

WEB4J provides the TimeSource interface to help you implement a fake system clock. When you define such a TimeSource, you are both defining a fake system clock for your application, and you are sharing your fake system clock with WEB4J framework classes. Thus, your application and the framework can use exactly the same clock, and will remain synchronized.

In addition, this same mechanism allows the default logging mechanism to timestamp records with your fake system clock time. See javapractices.com for an example.

Building New Apps From The Example App

The following is an outline of the recommended steps to follow to build a new application. The general idea is that the example application coexists with your app until it's no longer needed.

Phase 1 - Get Example App Running Against A New Database

1 - Download the latest version of fish.zip. Get it running, using the Getting Started Guide. Make sure the application can be launched and run before proceeding further. Note that .class files are included with fish.zip, so it's not necessary (or recommended) at this stage to compile the source. After each of the remaining steps, it's a good idea to verify that the application remains unbroken for each completed step.

2 - Change the base package/directory from 'hirondelle.fish' to the appropriate value for your new app. The supplied build.xml has an ANT target to help you:

• in build.xml, set 'replacement.base.package' to the desired value
• from the root/base directory, run the supplied ANT script : 'ant update.base.package' This ant script changes import statements and packages names.
• review and update directory names under WEB-INF/classes, if needed
• make sure you can compile and run the application
• (at this point, you might review and edit build.xml to suit your needs)

In web.xml, scan for settings which depend on the base package, and update them:

• the filter-class setting for the Preferences filter, and the File Upload filter
• LoggingLevels
• ImplicitMappingRemoveBasePackage
• LoggingDirectory
• 3 XXXDbConnectionString settings

3 - Create databases of the correct name.

There are two ways to update the database name:

• change the name of the already existing 'fish' databases (see MySQL docs)
• alter the script provided in WEB-INF/datastore/mysql/CreateALL.SQL, and run it.

At this stage, databases of the correct name exist, but their tables are temporary; the 'real' tables will be added later.

If MySQL is not available, then the database script will need to be ported to the target database. Although that may sound onerous, it's usually not that much work.

4 - Update the server configuration as needed. The connection pool and security realm info will need to reflect the new database names.

5 - edit hirondelle.config.web4j.AppInfo to reflect the correct information.

At the end of this phase :

• you have a new project directory, with code you can modify, with the correct base package
• you have a running application, talking to databases having the correct names
• the databases and application are simply the example app
• no 'real' features have been implemented yet

Phase 2 - Start Implementing Real Features

New tables should be added beside the existing ones. Do not remove the tables from the example app - not yet. In the beginning, your tables and the tables of the example application will coexist, side by side.

Modify Template.jsp, Footer.jsp, stylesheets, and so on, to create the basic look of your new application. It's often useful to create menu links at this stage as well.

New features (JSPs, code, and .sql files) should be placed beside existing features, in their own packages. Again, don't worry too much about removing existing items until later in development.

Usually, a feature will have these parts :

• a Java Server Page (JSP), for the view
• an .sql file, containing SQL statements
• an Action class
• a Model Object class
• a Data Access Object (DAO) class

Phase 3 - Remove Unnecessary Items

In most cases, removing a feature reduces to two simple actions : removing a single directory, and removing a menu link from some related template JSP.

Build Your Own Example App

For instance, the example app places user access tables in a database separate from the main business domain. Your app design may call for these items to be placed in one and the same database. Or, it may call for JNDI lookup to be used instead.

Recommendation - Often, your redesign of the example app can be retained for future use, as a your own example app, whose design is more appropriate to the needs of your organization. This will cut down significantly on repeated work.

Version History

• WEB4J
• the Fish and Chips Club example application
• the Predictions example application

The version numbers of the example apps reflects the version of web4j.jar with which it was built. For example, version 3.0.0.1 of the example app was built with version 3.0.0 of web4j.jar.