Bit depth
Concept
Bit depth (in terms of textures) or color depth determines how many distinct values you can store per channel (a channel refers to red, green, blue, alpha, etc) in an image. A .jpg image, for example, is an 8-bit format and can store more than 16.7 million different colors. So, what does that mean? It means each channel in a jpg can store a value that fits into 8 bits which can hold values between 0 and 255. Since a jpg only supports 3 channels (for red green and blue) you can store up to 256 x 256 x 256 = 16.777.216 colors. This sounds more than enough for anything, but keep in mind that black-and-white pictures thus only have 256 distinct values available - since every channel is exactly the same.
Bits per channel
It's worth mentioning that there are plenty of different image formats supporting different bit depths. The most common ones are 8, 10, 16 and 32 bit and formats may support one or multiple of them. However, keep in mind that these values refer to bits per channel! Some save dialogs may show you options that let you chose between 24 and 32 bit (for example when saving .tga); these are combined values and refer to how many channels you want to save. 24 bit is r,g,b or 8+8+8 bits while 32 bit refers to r,g,b,a or 8+8+8+8 bits. So the main difference here is the alpha channel which you may want to save or omit. Just be careful to not mistake it for an actual 32 bit per channel image.
Integer vs. floating point
You will be used to having color values in the 0-255 range (for 8 bit) or 0-65535 range (for 16 bit). These are integer color values and, as the name suggests, save values as integers. However, there are also floating point image formats (like .exr) which allow you to save values as floating point numbers - usually in either 16 or 32 bit float. This means you can save values like 0.35 or 287.878. In this case, the bit depth is an indicator for the precision, or the number of digits behind the dot you can save, and the maximum value. So the bits are split and used for an exponent and the fraction. Plus, depending on whether you chose to use a format that supports negative numbers, a bit for the sign.
Floating point formats have the advantage that they can store values above white, meaning values above 1.0. Integer formats, like a 16 bit integer format, usually have lots of precision but are limited to values between black and white, which is not always that useful. You can also save negative values which is great for displacements.
When should I use what?
8 bit integer:
These formats are great for the general use-case where colors aren't modified much and the precision of the stored value is not that important. Diffuse textures, for example, are generally stored as 8 bit per channel. However, as soon as you need to modify your colors it may be worth considering using a higher bit depth image format. For example, an 8 bit render output may suffer dramatically in post if you need to gamma-correct or do strong color corrections and grading on it. You will find that you're getting all sorts of issue ranging from color banding to strange saturation artifacts, especially in the lower ranges.
Also, for highly technical textures like displacement maps, 8 bit precision may have devastating effects on your render. An 8 bit displacement map can only store 256 distinct heights, so smooth gradients will inevitably degrade to staircases. Maps like roughness or metalness will usually work fine in 8 bit, Normal maps, however, can work with 8 bit but you may run into issues which are described in the "Dithering" chapter.
16 bit integer: 16 bit integer can be a suitable choice for information that has a known minimum and maximum (of black and white) and needs higher precision than 8 bit. Displacement maps and especially normal maps where the range is limited are good candidates. However, even in these cases, 16 bit float formats offer several advantages which may make them a better choice.
16 bit float: Since the introduction of OpenEXR (.exr), 16 bit float has been the widely adopted go-to format for render output and compositing. The advantages of ranges below 0 and above 1.0 make it a great choice for any information that may not fall into the standard black-to-white range. Also, due to its relatively young age, many compression algorithms are supported, which make exrs and thus 16 bit float a great and comparably lightweight choice for many tasks. Keep in mind, however, that keeping 16 bit information in RAM (for example in rendering) still uses the full uncompressed space, so this is not always the best choice. However, in terms of texture map types, displacement maps are a great use case for 16 bit float, as well as high precision normal maps that need smooth gradients.
32 bit float: This is more of an edge case and you want to be careful to only use it when it's absolutely necessary. The amount of disk (and memory) space it requires is hard to justify for any regular texture. It is mostly used to save specific render elements in offline rendering. For example, world position data - which contains the exact position of a pixel in 3D space - requires a very high amount of precision and can easily surpass 16 bit float formats.