Software Testing techniques — Baris Beizer, Dreamtech, second edition. Software Testing Tools — Dr. Prasad, Dreamtech. Follow us on Facebook and Support us with your Like. Average rating 4. Vote count: Modern Web Penetration Testing From Hacking to Report Writing.
Python Web Penetration Testing Cookbook. CompTIA Cybersecurit Wireshark for Security Professionals Cyber-Physical Attack Recover. Practical Information Security Management Phishing Dark Waters. Network Attacks and Exploitation. A Hacker. Hacker School. Automated Credit Card Fraud. Cracking Passwords Guide. Metasploit Toolkit — Presentation. Metasploit Toolkit — Syngress. Oracle Rootkits 2. Practical Malware Analysis. Return Oriented Programming.
Web App Hacking Hackers Handbook. Stack Smashing. When you view the program logic, it becomes clear that a small number of test cases will exercise all the possible combinations of paths through the code.
If you find you can't test anything this way, that all of your functions depend on all the other functions, you've got a serious problem on your hands. However, if you've written really clean, separable that is, object-oriented style code, you will find almost all of your functions are testable in this manner.
So you have this giant library of code, and you don't want to write test routines for all of it. So don't. You can write test routines for sets of 15 to 40 similar functions for example, strcat, strcpy, str in about three days. You will find few bugs, if any. However, those you find will be bugs that in the past you were never able to reproduce. And if you find no bugs, assuming you wrote good test programs, you now KNOW that your low-level functions are solid.
If most of your code is testable this way, you might be surprised by how few bugs make it into the final product if you test the pieces beforehand as described here. It may be coincidence, but when I started all-out testing of my low level functions is when I saw the biggest decrement in bugs in the programs I wrote.
Not only that, but subsequent to doing this, I cannot recall a single instance where a bug that seriously affected the entire program was found. Debugging in conditional defines To maintain speed, most times you will not want all of your conditional error checking to be compiled. Some of the error checking can take an extremely long time.
Generally, a specific check does not take a lot of time; rather, the check is called often. The other general reason is if you are printing debug information to a terminal or second monitor, and you don't want useful information to be scrolled off the screen by other information you have no interest in. Sometimes, the pure volume of printing can slow down the program because a monitor can write to the screen only at a certain pace, and no faster.
You can take either of two primary approaches to this problem. The first is to not compile in the extra debug code. This is much faster because the debug code flat out doesn't exist, but you will have to recompile if you decide you need some of the debugging prints back. The program example below illustrates the use of conditional compiling. The second approach is to test a flag at the beginning of the debug code and skip it if the test fails. Using the variable The program example below illustrates the use of a debug variable.
Although there are a lot of fancy ways to implement this conditional debug code, two rules seem to hold. First, you generally don't ever want to turn debugging on or off after a compile. Second, both ways of implementing this seems to be equally fast to code and execute. In these cases, using ifdef debug alone is generally sufficient, and anything more is a waste of time.
For code that is otherwise simple but has a couple of time-consuming checks, you can use the value for which DEBUG is defined as conditional compiles. Keep in mind with conditional debugging that some debug tests are matched. If your allocate conditionally sets the signature bytes before and after each region of allocated memory, it better do it if any of the conditional defines that check the heap for the signature bytes are executed.
I once spent several hours trying to determine why some buffers were failing their CRC tests - the reason was the CRC creation code had been ifdef'ed out while the checking hadn't. Comment your open Issues As you code, issues or questions come up in certain pieces of the code: Will a function work on a Super VGA monitor? How do you handle data files with a particular error?
Do you need to watch out for a certain error condition? These questions are not resolved by Traps and IntTests. Rather, they are items that you believe you have not fully coded for or, in some cases, you have not coded for at all. In other cases, your program will fail under conditions in which you need it to continue running.
These are possible logic errors that occured to you at one time or another. Unfortunately, the human mind being what it is, these errors will be quickly forgotten until the bug report comes in. The place to write down these concerns is in the code itself.
Anywhere else and the place it is written can get lost. If its in the code - its there forever, or until you resolve the question.
In these cases, place a one-line comment, the word "bugbug" followed by your initials and the date. You can follow this with a comment that runs for several lines, but the first line should give a reasonable description. Should we go to disk? Unfortunately c doesn't support printing during a compile in a manner that makes this possible. You can't lose the list of open items unless you lose the code too.
In the above example, you might have decided to go to disk. You might also have decided you could limit yourself to memory. In either case, the question was resolved. You might not remember the issue until users report the bug after the program. Keep it unobtrusive Keep your debugging as unobtrusive as possible. The rest of your program should hardly be affected by debug code. The program might run a tad slower, and the user might see a pop-up debug message. Good debug checking stays hidden and out of the way until it finds a MFU.
Now we get into the actual programming tricks. For those of you who have no idea how to program, if you're still with us, its time to skip to the chapters on testing.
For those of you in management, its time to give your developers a raise and then skip to the testing chapters. This chapter has all the tricks that don't deserve a chapter of their own. This isn't to say these tricks aren't important, merely that they can be explained in fewer words. This, and the rest of the chapters until we get to the testing process, both lists specific tricks and how to implement them. The tricks are described generically. The code is c and assembler where required code for the PC.
Most important, these tricks will allow you to painlessly eliminate bugs from your programs. These tricks do not add to your work - but they do save a lot of the time you previously spent debugging. TestAll functions Suppose your program is mysteriously crashing. You don't know why or exactly where. But then you place the TestAll function at various points in your program.
You are now able to narrow down the bug. TestAll reports all is okay and then, 20 lines later, reports an error. The bug is in those 20 lines. So what is this magical TestAll function? Quite simply, it calls every global test function you have. At a minimum, it checks the NULL memory locations, the entire heap, and the amount of stack left. It also asserts all global data structures.
Anytime you create a test function that can be called anywhere globally to perform its test, that test should be added to TestAll. Eventually, TestAll becomes very slow, and because of that you rarely want to call it.
However, it proves invaluable in two places. It is invaluable because you are switching data sets and you should know which data set doesn't work. It is a quick way to identify the bug. You place the TestAll function at various places— where you know the system is still okay up to the point where you know it is trashed. It will not only find where you first start to mess up your program; it will also tell you which test failed, helping you identify what is wrong, and where.
In the below example, funcBad blows up due to an error in the system. But the code looks error-free. Before walking through funcBad, you can see if func1 or func2 is actually the culprit although you don't see the bug's effects until funcBad. Restoring system state We have all used programs where the program itself runs fine, but DOS always seems to lock up shortly after using the program.
There is one program I used to use regularly with no bad affects ever. Obviously, it was playing with memory that didn't belong to it. When your program is about to exit, you should fully test the system. Many programmers don't worry about an error message on exit, thinking, after all, you are exiting—who cares if things are a little screwed up? This is a very dangerous practice.
First of all, many of these errors are symptoms of bugs that could, under different configurations or use, cause problems while you run the program. Second, many of these errors are damaging to the operating system. Be careful to catch a program's exit under all conditions. Call TestAll. This single call should test the integrity of your system. Call the StackUsed function discussed later.
This call will tell you the maximum amount of stack space used. Every time you exit the program, print out the maximum stack size used as well as the total stack size.
This gives you constant feedback, letting you know whether you have set your stack size appropriately. This is not a concern under Windows unless you are writing some VERY dirty code—in which case this is the least of your problems.
Bounds Checker discussed in the tools chapter will check the interrupt table as well as the BIOS data area on exit to see if you have changed any of it, so there is no sample code here for this. However, if you don't use Bounds Checker, you need to write a program that saves the interrupt table on start-up before the c-run-time initialization code runs.
Leave the system in the state in which you found it. This is critical to insure that your program does not swallow up resources every time you run it. Any files opened should be closed. Also, if this memory is not freed, your program is probably wasting resources while it is running. Check your heap for allocateated memory that is not freed.
If you use both the near and far heaps, check both. Unfreed memory generally indicates a logic error when you finish an action. Even if the only problem was that you forgot to free a pointer, you are still slowly eating up a scarce resource. Check for files that are still open. This might be indicative of a logic error when you finish an action. However, open file handles, as well as being a scarce resource like the heap, are much more dangerous if left open.
Second, if another application wants to access the file and you have opened it with exclusive access, you have locked out the other application. If you are running Windows, insure that you have freed up all resources. Some Windows resources are still in very short supply for example, Windows has only five screen-device contexts. Most of the examples in the list above are illustrated later in this book.
However, some programs use resources that are not as general. Be sure that you have freed up all those resources. Your mother was right, it is very important to clean up after yourself. Essentially, it gives you a running list of what function you are in and where you were called from.
It can also be used by a debugger to track your call stack. This allows the debugger to list,by function name, the calls you made beginning with main to get to your present location. The EXIT macro prints that the program is leaving the function and returns from the function.
These are constants defined by the compiler. Using these constants, you can give the source location of a debug printf. However, this method leaves a lot to be desired. First, where were you called from? Second, a given function might call strcpy from multiple places, some of which might be in loops. Where was a specific strcpy called from? Finally, if a function exits in multiple places, you have to place the EXIT macro in all of those places.
A better alternative is to create a macro for calling a function. The macro will then call the function. Finally, it will print that the function returned, optionally including the return value. However, most of the C compiler vendors are putting the capabilities required for this scoping in their C compiler also.
Functions can track how deep the function is nested and indent the printfs appropriately. Regardless of how you set this up, be sure you don't drown in data. Also, if the logging occurs on every function call, including those to the low-level library calls, your program can slow to a crawl as it spends most of its time spewing vast amounts of data to the debug terminal, virtually all of which is ignored.
So when should you use this? I list it because it is a technique some people use. Error Message file In most cases you will probably want to send your debug output to a second monitor. In some cases, you might want to send it to a log file. The easiest way to do this is to have the DebugPrintf decide whether printfs are going to the monitor, the file, or both. The error file needs to be opened when the program first starts. You will probably want to either create the file deleting the previous log file or use a new name each time such as incrementing the extension until you find a filename not in existence so that you are not creating a very large file.
If you don't do this and the program locks up, you will lose the most recent writes, usually the ones that tell you why the program locked up.
However, a commit on each write is slow. And you can't use your RAM disk— that is lost if the system locks up. I have found that writing to a log file, with enough information and commits to be useful, is so slow as to be useless. If you do choose to use an error log, under DOS 4. Use this: It's lots faster than calling commit file DOS 3. Do this under only DOS 4. Object-oriented programming Object-oriented orogramming OOP is the buzzword of the moment.
It will solve all your programming problems, give you unlimited wealth, and make you irresistible to the opposite sex actually, unlimited wealth will do that too. The subject of OOP merits a book of its own. In this book, we will discuss how OOP can help stop bugs from ever showing up. It's also important to discuss just what parts of OOP give you these advantages.
OOP does not require any specific language. You can write OOP code with assembly language. At the simplest level, OOP means keeping your function separate. For example, functions that handle circles have no code or data in them for squares. Code for squares has no code or data for circles.
Code for drawing an item has no code or data for circles or squares—but knows how to call the circle or square drawing functions. This separability in your program makes it easy to implement exhaustive tests for virtually all the functions in your objects. This separability also means that a change in one object doesn't affect any other object.
Use an OOP approach to your coding. It will pay off in a much-easier-to-test product. That said, realize that OOP is not a panacea.
It makes it easier to do some kinds of testing but it does not eliminate the need for any tests. It also adds some new bugs that you normally don't see in c or asm. Filling buffers Another bug that makes its presence rarely felt is using buffers before they have been initialized or after they are free'ed.
When you do so, in most cases, the buffer will still have the correct data. Many other times the buffer will have legitimate data, although not the data it should - leading to very subtle bugs. Whenever a buffer is considered empty, fill it with an ID byte or string if you prefer. This ID byte should be a value that is invalid for the structure being filled -2 usually works for everything.
This way, if you access the buffer after filling it, you will get data that, hopefully, is so incorrect, that it causes an immediately visible MFU. This is very valuable before file reads to insure that you handle the end of file when you get a partial read successfully. Filling passed in buffers is also a good check to make sure the passed in buffers are long enough.
If a function returns a string, and one of the parameters is the maximum length of the string, filling the buffer first will insure bugs if the passed in buffer is too short. Also, when you free up a buffer, first fill it.
This prevents bugs due to accessing a free'ed buffer which usually holds the old data. This holds not just for freeing a buffer you allocateated but also for internal cache buffers, data structures, and so on. Anytime you are through with any kind of memory, fill it before freeing it. By filling freed memory, then if you access this memory after the free, you are guaranteed to get bad values, hopefully causing an obvious MFU.
If you don't fill it, then when you access the memory after it is freed, you will see the bug only in the rare cases where the memory is reused prior to your post-free access. In some circumstances 0 is deadly; in others, it's totally benign. Test your entire program using 0x00, 0x7E, and 0xFE.
I have found 0xFE to be the most damaging in most cases. I don't use 0xFF because -1 in many cases has a special, sometimes benign, meaning. If you delete a record in a database file, filling the deleted record will guarantee that you get bad data if another record is still pointing to the deleted record. Before any read, not just from disk but from anywhere RS port, and so on , fill the buffer with the ID value. Then, if the read returns with no error but didn't actually read anything, the buffer will not have it previous contents, possibly from a previous successful read.
If the read is partially successful that is, you read the end of a file , the fill performs a second function. If you assume all reads are fully successful, you will get bad data. Otherwise, you are usually using the same buffer to perform all reads, and the remainder of the buffer generally holds what appears to be legitimate data.
You don't need to do this prior to a MemCpy or other direct memory copying. However, whenever you expect someone else to fill a buffer for you DDE, networks, and so on , fill the buffer first. Filling Buffers The program example below illustrates the use of filling a buffer prior to a read.
I have found it useful to set registers that I use in assembly language modules to this same ID value. In higher-level languages you generally don't have scratch global variables you can, but it's a terrible practice. The idea is to fill any variable of any type before someone else will set it or after you are done with it.
This is one of the few things that the compiler companies have actually tried to do something about themselves. In any event, watch for NULL pointer writes. For other processors, you might still have more than one NULL location. Regardless of the quantity of them, copy the 4 four bytes from each NULL to a global variable on start-up. Then when the program ends, compare these values to the values at each NULL location.
Write a message to the tester if they don't match. Even though most compilers perform this checking for you, do it yourself for two reasons. First, most of the compilers test only for overwriting SS Second, you should have a TestNull function to add to the TestAll function.
Keep the following thoughts in mind. First, sometimes the NULL value will legitimately change. If you set an address on int 0, the NULL changes.
Also, some compilers set DS:0 to 0 and then use that pointer for literal strings that are empty. Therefore, writing to DS:0 can cause some unpredictable behavior. I would spend hours tracking it down. And then I would finally find it. I would have done something like pass an int when I was supposed to pass a pointer to an int. I did this so often, that as soon as the compilers started doing prototype checking I'm showing my age here , I always prototyped my functions.
For a very small effort you get a very large return on your investment. C is designed to help you do anything you want. However, it also has the ability to use function prototypes and perform very strong type checking. Use prototyping. A good C programmer breaks the rules at times—carefully and deliberately. All functions should be fully prototyped.
This includes using const whenever possible on the prototype for function variables. Prototyping examples The program example below illustrates the use of proper prototyping:.
All code should compile at the highest warning level with NO warnings or errors produced. With C's casting capabilities, there is no need for any warnings.
This full prototyping has two advantages. First, if you see a warning, you know something is wrong although it might be as simple as having forgotten to cast something. Second, this will catch a number of errors that otherwise might not be seen for some time and that could take days to resolve.
Prototyping rarely will show me where I have made a mistake, but when it does, it generally shows the type of bug that doesn't normally make itself known till much later, When it is very difficult to track down. While most warnings can be resolved with a cast, that should be your last response. I once found a bug in someone's code where they passed a long not a pointer to a long where a function wanted a far pointer. Needless to say, this caused the function to bomb. When asked about casting the long to a pointer, the programmer replied, "To eliminate the warning message.
Unless its clear to you why you do need to cast, you are, at a minimum, doing something you don't fully understand. A blow torch is a valuable and necessary tool for building cars. However, it is not a recommended tool for repairing a circuit board. Effective use of tools requires knowing when to use and when not to use something.
It's very important to know when the solution to a warning or error isn't casting but is instead correcting your code. However, a developer I respect a lot swears by them so I leave it up to you. I am sure there are times it can pay off. The first place to use CRC checking is on data structures especially those buffering generic data. By performing a CRC check, you gain two additional checks. First, if there is no way to make a consistence check the buffer is from reading a binary file - any values are legit of the buffer, we can still verify that the data is good.
Second, if we want to make sure the structure hasn't changed to a different set of consistent data, the CRC will tell us if its the data set we believe, even if the data passes our consistency test. If you use Bounds Checker see the tools chapter or you are running under a protected mode operating system such as a Windows application in standard or enhanced mode , it will do this for you.
However, if you don't have these options, you can perform a CRC of these segments on. In both cases, this is usually due to a loose pointer. Keep in mind that some third-party libraries use self-modifying code which would make this test difficult.
However, with the chip pre-fetch queue in the and beyond, self-modifying code has become a dangerous practice that is being used less and less. Roll your Own Keep in mind that this chapter is filled with suggestions, not a rigid set of practices to take as a whole.
Some of these will work better for you, some worse. You want to use those that will pay off for you and not waste your time with those that won't. In addition, if you have practices of your own that are not mentioned here, continue to use them. The trick is to use what works, not what is written down. These tricks are ones to add to your repotoir, not to replace it.
A couple of years ago, I was writing a piece of code where I was performing some very complicated actions on a buffer I was walking. I had several pointers, that had a very specific relationship to each other. If one of these pointers was off in relation to the others, then I would get major errors. I spent days on this code, each time I found a bug, tracking down its cause. Almost every time I would fix one bug, another would get created.
Finally, in desperation, I wrote a function to check the pointers relationship to each other at the begining of each pass through the loop. If the pointers didn't match, I printed out the value of every local variable.
Within a couple of hours, I found the remaining bugs and had the function working. Subsequent bugs in the code basically identified themselves.
Since then, I have come to use this practice assertion virtually everywhere. Not only does it save you days of tracking down bugs, but it finds bugs you otherwise wouldn't find.
And finally, for the asserts that never find a bug, you can feel that much more secure about your code - you have parts that you know are working correctly. As you go through your code, you have places where you know that a variable holds a certain value and you have places where you believe that a variable holds a certain value.
And unfortunately, you might have places where you hope that a variable holds a certain value major MFU alert.
In most cases, the honest answer is that you believe a variable holds a certain value. The fact that you don't know is part of programming. If you write a low-level function, you have no control over the code calling your function.
In a complicated procedure you believe that you understand every possible path the code can take but you can't know for certain. Assert to the rescue To solve this, we make massive use of the assert capability. An assert function checks a Boolean expression and, if it is true, calls the warning message function. If a string is ever longer than 15 bytes, the assert macro will print a message listing the file and the line in the source code that caused the error.
Most compilers support two asserts. The second assert is placed only where you want the retail version of the program to exit if the Boolean value is TRUE. This is bad coding—you have left MFUs in your program. There should be no assert in non-debug code. Under certain conditions, you might want to end your program. However, you should exit gracefully with a message to the user that makes sense. How good will you feel about a program that suddenly exits to DOS with the line internal error dave.
This bears repeating because there are numerous commercial programs that ship with asserts in them. You need to handle ALL error conditions properly. Exiting to DOS with an incomprehensible message to the user is not acceptable. Give the user enough information that, if its in the morning and they need to use your program to complete something by in the morning, they have a prayer of figuring out what to do differently so the program will work. You assume certain things can't happen or must happen at certain places in your code.
Place an assertion in your code that tests for this and prints if the impossible happens or the certain fails.
The best way to place these is to look at your functions after you have finished coding them and ask yourself what can go wrong. Usually one or more places in the code will jump out at you.
Place asserts at these spots. While debugging you will also come across bugs that would have been solved by an assert. Place one there and remember the situation the next time you are placing asserts. As you walk through a tree, if you assume a certain pointer is always good, then assert pTree! One of the most important assertions is that you check the ID in the struct when a function is passed, or returns a struct. Assertions are usually a low-investment, high-return effort.
For any given data structure, you need to write an assertion function. Then, anywhere you believe you have a pointer to the structure or a variable holding the structure, but you aren't percent sure, assert the variable. Generally the code in an assertion is quite fast, so load up your code with assertions everywhere.
You won't see much of a hit in speed. But when an assertion fails, you are pointed right at the failure, by module name and line number. We break assertions into 2 parts; the code to check a data structure and the code common to all assertions.
We will discuss the common code first. C has had an assertion macro for years. If false, it will print out the false statement, and the module and line number at which it occurred. We will get a little bit more sophisticated than that. Each structure has its own Check function. You can use an assert macro, but then all the checking occurs in-line.
This not only leads to a bigger program, but it also makes complicated checking impossible. If there is a problem in the Check function, the Check function calls a standard AssertPrintf. This way all assert printfs begin the same way. For example, in Windows, you use a MessageBox and the caption can then be Assert - file: dave.
The rest of the printf can be any formatted string the Check function wants to display. The Check function should say what it didn't like but don't worry about making it pretty. Ok, we can print an assertion failure, but how do we know if we actually do have a problem? The first part of the Check function is the code to check the data structure.
Are the values consistent with each other? Sometimes you can't truly tell whether you are pointing at the data structure, so all you can do is see whether it holds valid data. So you see how we can check the big, complicated structures in your program. But what about simple data? Most code has a lot more ints and strings than structs. All you need to do is take a look at what you are using the variable for. To demonstrate, in the following section we will create an assert for a bool, int, and string.
AssertBool You can assert a Boolean value very simply. TRUE is defined as a non-zero value, but in virtually every program written it has a specific value, usually 1 or If a Boolean is an integer, the odds of randomly holding a legit value are even worse. A Boolean with a random value will almost always fail this test.
0コメント