Content and Privacy Protection in JPEG Images by Reversible Visual Transformation

Abstract

With the popularity of cloud computing and social networks, more and more JPEG images are stored and distributed. Consequently, how to protect privacy and content in JPEG images has become an important issue. Although traditional encryption schemes can be employed, the file format of JPEG images is changed so that their usage may be affected. In this paper, a reversible visual transformation algorithm is proposed to protect content in JPEG images. Specifically, the DC coefficient in each user-selected block is modified, while the information required to recover it is reversibly hidden into AC coefficients. Then the signs of AC coefficients in the selected blocks are flipped and the blocks are further scrambled with a secret key. By embedding the location information of the selected blocks in a transformed image, the original image can be exactly recovered when needed. Besides, regions to be protected can be arbitrarily chosen without substantially affecting the rest of the image. The experimental results on a set of JPEG images validate the efficacy and reversibility of the proposed algorithm. In addition, good performance is achieved in terms of invisibility of the protected content, image quality, file size preservation and security.

1. Introduction

With the popularity of cloud computing and social networks, more and more images are stored and distributed on the web. There are quite a few image formats, among which a popular one is JPEG [1] for its high efficiency in compression. Compared with lossless formats such as bitmap (BMP), JPEG compression can significantly reduce the file size to save bandwidth without introducing noticeable degradation. With the dissemination of JPEG images, how to protect content and privacy for them has become an important issue. For instance, a user may want to share some JPEG images containing private information but directly exposing them in social networks may impair privacy.

In the literature, a number of methods (e.g., [2,3,4,5,6,7,8,9,10,11,12,13]) have been proposed for privacy protection in JPEG images, including selective encryption (e.g., [2]), image blurring, image pixelation, image masking, image morphing (e.g., [3]) and image warping (e.g., [4]). To keep the sensitive content from being exposed, these methods may change image content permanently and even degrade image quality. By contrast, it is advantageous for authorized receivers to recover the original images when needed. Hence, data hiding has been applied in JPEG image encryption to achieve such reversibility. For instance, scrambling and image data hiding are combined in Ong’s algorithm [5]. Moreover, image masking and data hiding are combined in [6,7]. Nevertheless, the original image cannot be completely recovered with the algorithms proposed in [5,6,7]. Then, a privacy protection algorithm based on scrambling the quantized Discrete Cosine Transform (DCT) coefficients was proposed in [8], but the JPEG file size is significantly increased. Reversible data hiding (RDH) (e.g., [14]) is applied in Joshi’s algorithm [9] and Niimi’s algorithm [10] so that the original JPEG images can be completely recovered. In particular, Joshi’s algorithm uses a prediction-based RDH technique to achieve reversibility and protects sensitive regions by modifying pixel values in them. However, the size of the privacy-sensitive region (PSR) is restricted. Additionally, a large PSR may cause distortions of a non-sensitive region while a small PSR may increase the file size. In [10], Coltuc’s reversible watermarking algorithm [14] is adopted to modify AC coefficients for content protection. Subsequently, two successive AC coefficients are further modified to mark the protected blocks. Although the performance of [10] for content protection was evaluated, the security has not been properly analyzed. As a matter of fact, the protected blocks can be easily identified and the AC coefficients may be restored by a brute force search.

To enhance the security, we improve the algorithm in [10] in two aspects. Firstly, the location information of protected blocks is generated instead of modifying two successive AC coefficients. By embedding the location information into the whole JPEG image with the RDH algorithm in [15], the protected blocks can be identified only after data extraction. Secondly, the protected blocks are scrambled with a secret key in our proposed algorithm so that it is more difficult for an adversary to recover the original image. Similar to the algorithm in [10], the information required for image recovery is reversibly hidden in the AC coefficients by adopting the algorithm in [16]. It turns out that the original images can only be obtained after extracting the hidden data with the secret key.

To evaluate the performance, invisibility of the protected content, image quality, reversibility and JPEG file size increment are measured. The experimental results show that the proposed algorithm provides good security against attacks. The protected content is invisible while the quality of the other blocks is well preserved. Moreover, the proposed algorithm outperforms the existing techniques in terms of reversibility and file size preservation. In the next section, some related work will be reviewed. Section 3 introduces details of the proposed algorithm. Section 4 presents the experiments and evaluation results. Finally, we draw conclusions in Section 5.

2. Related Work

The algorithms for privacy protection in JPEG images can be classified into two categories: (1) algorithms performed on bit stream (e.g., [2]), and (2) algorithms performed on the quantified DCT coefficients (e.g., [5,6,7,8,9,10]). Since modifications of the code stream may change the format of compressed images, the decryption should be completed before decoding and a dedicated decoder needs to be equipped. By contrast, algorithms processed on the quantified DCT coefficients are more flexible and protected images can be decoded directly.

Advanced encryption standard (AES) [17] is a classical and popular symmetric encryption algorithm, which was applied in a selective encryption of JPEG2000 in [2]. In the selective encryption algorithm, the bit stream is divided into different packets, and the most significant part of the entire packet data is encrypted by AES. This algorithm works effectively on JPEG2000, but it cannot be applied to JPEG images because of the different coding schemes. Nevertheless, AES can be applied in the encryption of JPEG images in another manner. The encryption is achieved by simply recognizing the JPEG code stream as common plaintext without focusing on the characteristics of JPEG. A series of operations such as byte substitution, row displacement, column mixing, round key addition and exclusive or are conducted on the plaintext. However, the visibility and the file size of JPEG images will be affected. Besides, it requires more time to encrypt an image with AES because of the complicated calculation process.

Alternatively, quite a few algorithms have been proposed by utilizing the characteristics of the JPEG coding scheme to achieve privacy protection. These algorithms are processed on the quantified DCT coefficients, and the proposed algorithm also belongs to this category. Specifically, the file size increment should be controlled within an acceptable range while achieving content protection and security against attacks.

The P3 algorithm [11] leverages sparsity and the high quality of JPEG images to divide a JPEG image into a public part and a secret part. The public part is exposed to the photo sharing service providers, while the secret part is encrypted and shared between the sender and the recipients. As a result, the two parts need to be stored separately, which introduces an extra cloud storage request for the secret part and complicates the file management system. In addition, the P3 algorithm only works on the whole-image level and it is inconvenient for users to select the protected regions.

Yuan’s algorithm [12] allows users to select several regions of arbitrary shapes to be protected. The signs of the DCT coefficients in selected regions are modified to scramble the image and the auxiliary information for recovery is inserted into one or more application markers in the JPEG file header. Thus the privacy and its reversibility can be achieved. The data hiding approach is employed to avoid extra overhead to storage, but it causes a larger file increments compared with other algorithms.

Both Li’s algorithm [7] and the algorithm based on false colors [18] modify the value of DCT coefficients to protect the sensitive information and record the difference for recovering. In Li’s algorithm, a cartoon mask is used to cover a human face by calculating the DCT coefficients of the mask and replacing the original ones. The original value of the DCT coefficients and the difference are embedded in the JPEG image. In the process, the sparse representation is used to find a more concise expression for each original image signal, so the cost of file storage can be decreased. Nevertheless, F5 steganography [19] or another JPEG steganography algorithm (e.g., [20]) is applied in the embedding operation, but the original JPEG images cannot be recovered completely. Similarly, a scalable scrambling algorithm operating in the DCT domain is proposed in [13] within the JPEG codec. The goal is to ensure that people are no more identifiable while keeping their actions still understandable regardless of the image size.

The algorithm proposed in [18] substitutes false color for each original color value through a specific mapping relationship. As a result, detailed information, such as a human face, cannot be identified. Since the algorithm exploits the coherence of the false color version of the encrypted JPEG images and the original ones, the size of the protected content is reduced. Moreover, lossless image recovery is achieved since the difference and other auxiliary information is hidden in the JPEG image to be protected. However, the embedding operation may lead to a large cost of file storage. Since the privacy protection based on false colors is mainly used in video surveillance, the image outline is visible and some privacy information is exposed.

As the existing algorithms cannot fully meet the requirements of privacy protection in JPEG images, especially in reversibility and file size preservation, a reversible and effective algorithm is proposed in the following section, which achieves not only reversibility but also good performance in terms of invisibility of the protected content, image quality, file size preservation and security.

3. Proposed Algorithm for Content Protection in JPEG Images

The algorithm to be presented can be used to protect both grayscale and color JPEG images. The algorithm for grayscale images will be introduced only, which can be directly applied to the luminance component of a color image. The proposed algorithm consists of two parts, i.e., converting an original JPEG image into a visually transformed one and recovering the original JPEG image. In the following section, the two parts will be introduced in detail.

3.1. Generating the Visually Transformed JPEG Images

The flowchart of generating the visually transformed JPEG image from an original one is shown in Figure 1, in which user interactions are needed to select the regions to be protected. According to the user selection, the image blocks containing the selected regions can be identified and their positions can be recorded as binary location information. Then DC coefficients in the selected blocks are modified while the difference between the original and modified DC coefficients is reversibly embedded into the AC coefficients. After that, the signs of the AC coefficients in the selected blocks are flipped with a sequence of 1 s and −1 s generated with a secret key. The selected blocks are further scrambled but the DC coefficients in them are kept unmoved. Finally, the protected JPEG image is generated by reversibly embedding the location information into it.

3.1.1. Key Generation

For security concern, a secret key is used in the proposed visual transformation algorithm. The secret key is shared between the sender and receiver, or can be transmitted by using public-key cryptography. To generate different keys for different images to be protected, the secret key can be concatenated to the mega information of an individual image to generate a hash value of 128 bits by MD5 [21]. Then the hash value is used as the seed of a pseudo-random number generator (PRG) to generate random sequences in the flowing steps. In this way, the user can use the same key to obtain different seeds for different JPEG images to resist brute force attack [22].

3.1.2. User Interaction to Select the Regions to Be Protected

A JPEG image is divided into 8 × 8 blocks and each block is assigned with an index ranging from 0 to n − 1, as there are n blocks in the JPEG image. When a user selects one or more rectangular regions to be protected, the location information of the selected regions will be recorded. For each selected region, the index of top left block, and the height and width in blocks are recorded. Suppose there are w blocks per row and h blocks per column in a JPEG image with n blocks. The location information of each block consists of three parts with the lengths as follows:

(1)$l=\lceil {log}_{2}n\rceil$

(2)$l_1=\lceil {log}_{2}w\rceil$

(3)$l_2=\lceil {log}_{2}h\rceil$

where l, l₁, l₂ represent the lengths in bits for block index, region width and height, respectively. The recorded information of all selected regions is combined together to form a location map. Note that the length of the generated location map is calculated, which is denoted by len and embedded into the JPEG image prior to the location map.

3.1.3. Modifying the DC Coefficients

Differential pulse code modulation (DPCM) and Huffman coding are used to encode DC coefficients by exploiting the correlations between adjacent DC coefficients. To enhance the security while reducing the JPEG file size, all DC coefficients in selected regions are modified to similar values. For a DC coefficient $d$, the modification is conducted by

(4)$d^{\prime }=d-delt{a}^{\prime },$

where d′ represents the modified DC coefficient and $delt{a}^{\prime }$ is obtained as follows. At first, the difference between $d$ and the average (denoted by avg) of all DC coefficients in the selected blocks is calculated by

(5)$delta=d-avg,$

where delta is the original difference between $d$ and avg. To reduce the data for representation, delta is quantized to delta′ with the method mentioned in [23] by

(6)$delt{a}^{\prime }=\{\begin{array}{c}8\times round\left(\frac{delta}{8}\right),delta>0\\ 8\times floor\left(\frac{delta}{8}\right)+4,delta<0\end{array},$

where $delt{a}^{\prime }$ is the multiple of 4 so that it can be represented with fewer bits. For recovery, the value of delta′ is embedded into the AC coefficients in the same block by adopting an RDH algorithm proposed in [16]. Specifically, the binary value of $\frac{\left|delt{a}^{\prime }\right|}{4}$ is hidden in the AC coefficients and we can recover delta′ from $\frac{\left|delt{a}^{\prime }\right|}{4}$ by multiplying it with 4. In addition, the sign of $delta$ can be inferred from whether $\frac{\left|delt{a}^{\prime }\right|}{4}$ is odd or even. As the experimental results conducted in [24] show, hiding data in low-frequency coefficients helps to achieve a smaller file size, and the 3rd to 11th AC coefficients of each block are chosen to carry the bit values. Since each DCT coefficient ranges from −1023 to 1023, delta is within [−2046, 2046], which indicates that $\frac{\left|delt{a}^{\prime }\right|}{4}$ can be expressed with 9 bits. For each AC coefficient a_j (j = 3, 4, 5,…, 11) and a bit value s (0 or 1), data embedding is performed by

(7)$a_j{}^{\prime }=a_j\ast 2+s.$

Now we take the example as shown in Figure 2 to explain the aforementioned process. Suppose there are four blocks in the selected region and the DC coefficients are 37, 3, 20, 20 with an average value of 20. For the first DC coefficient, the value of delta is 17 and the value of delta is modified to 16 according to Equation (6). After obtaining the quantified delta, the DC coefficient is modified to 21 by calculating the difference between the original one and delta′. Finally, $\frac{\left|delt{a}^{\prime }\right|}{4}$ is represented in nine bits, which are embedded into nine AC coefficients of the same block with Equation (7). The cases of modifying the other DC coefficients are also shown in Figure 2.

3.1.4. Permutation of the Selected Blocks

A random sequence R1 of 1 s and −1 s is generated with a PRG, which takes a secret key as the seed. Then the signs of AC coefficients in the selected regions are flipped by multiplying the sequence of AC coefficients with R1, while the selected blocks are further scrambled with another random sequence, R2, generated with the same secret key. Specifically, the DC coefficients are kept unmoved and the lengths of the two random sequences depend on the number of selected blocks. Suppose there are n blocks in the selected region, the length of R1 should be n × 63 and R2 contains all integers between 1 and n. Then the JPEG image with the permuted blocks is generated.

3.1.5. Hiding Location Information in the Transformed Image

In this phase, histogram modification is applied to AC coefficients with value 1 or −1 for reversible data embedding (e.g., [15]). The other AC coefficients are modified by histogram shifting. The operation is to embed a bit value i into an original AC coefficient a_j by

(8)$a_j{}^{\prime }=\{\begin{array}{ll}0,\hfill & a_j=0\hfill \\ -1,\hfill & a_j=-1,i=0\hfill \\ -2,\hfill & a_j=-1,i=1\hfill \\ 1,\hfill & a_j=1,i=0\hfill \\ 2,\hfill & a_j=1,i=1\hfill \\ a_j+1,\hfill & a_j>1\hfill \\ a_j-1,\hfill & a_j<-1\hfill \end{array},$

where a_j^′ is the modified AC coefficient.

3.2. Recovering the Original JPEG Images

The flowchart of recovering the original JPEG image is illustrated in Figure 3. Note that the same key generation process as in the visual transformation process is conducted to obtain the seed of PRG to generate two random sequences. After extracting location information from the transformed JPEG image, positions of the selected blocks can be identified. Then the permuted blocks are moved back to their original positions according to the random sequence generated with the secret key. Meanwhile, another random sequence of 1 s and −1 s can be generated as in the visual transformation process so that the flipped AC coefficients can be restored. After that, the original DC coefficient in each block is obtained by extracting the difference value hidden in the AC coefficients. Thus, the original JPEG image is recovered.

3.2.1. Extracting Location Information

According to Equation (8), an AC coefficient contains an embedded bit only when it is ±1 or ±2. So, the coefficients with the embedded data can be distinguished from the others. A hidden bit is extracted from an AC coefficient, while the AC coefficients are restored by

(9)$i=\{\begin{array}{ll}1,\hfill & a_j{}^{\prime }=-2\hfill \\ 0,\hfill & a_j{}^{\prime }=-1\hfill \\ 0,\hfill & a_j{}^{\prime }=1\hfill \\ 1,\hfill & a_j{}^{\prime }=2\hfill \end{array},a_j=\{\begin{array}{ll}0,\hfill & a_j{}^{\prime }=0\hfill \\ -1,\hfill & a_j{}^{\prime }=-1,-2\hfill \\ 1,\hfill & a_j{}^{\prime }=1,2\hfill \\ a_j{}^{\prime }+1,\hfill & a_j{}^{\prime }<-2\hfill \\ a_j{}^{\prime }-1,\hfill & a_j{}^{\prime }>2\hfill \end{array}$

where i is an extracted bit value, a_j is a restored AC coefficient and a_j^′ is the corresponding AC coefficient in a visually transformed JPEG image. After data extraction, the values of l, l₁, l₂ in Equations (1)–(3) are calculated by collecting the extracted bits so that the positions of the protected regions can be known.

3.2.2. Recovering the AC Coefficients

According to the secret key, the same seed used in the visual transformation process can be generated and used as the input of PRG to generate the random sequences. Consequently, the location of each permuted block can be known so that each block is moved back to its original position. Note that DC coefficients in the permuted blocks are unmoved in the visual transformation process. Then the signs of all AC coefficients in the restored blocks are flipped with another random sequence of 1 s and −1 s so that the JPEG image with modified DC coefficients is obtained.

3.2.3. Recovering the Modified DC Coefficients

After recovering the AC coefficients, the data hidden in them (i.e., $\frac{\left|delt{a}^{\prime }\right|}{4}$ ) can be extracted to recover the original DC coefficients. For each block, the embedding data can be extracted by

(10)$a_j=\frac{a_j{}^{\prime }}{2}$

(11)$s=a_j{}^{\prime }mod2,$

while a_j^′ (j = 3, 4, 5,…, 11) is an AC coefficient carrying a bit value, a_j is the original AC coefficient and s is a bit value in $\frac{\left|delt{a}^{\prime }\right|}{4}$ of the DC coefficient. Then the difference between the modified DC coefficient and the original delta can be obtained by

(12)$delta=\{\begin{array}{c}\frac{\left|delt{a}^{\prime }\right|}{4}\times 4,if\frac{\left|delt{a}^{\prime }\right|}{4}mod2=0\\ -\frac{\left|delt{a}^{\prime }\right|}{4}\times 4,if\frac{\left|delt{a}^{\prime }\right|}{4}mod2=1\end{array}.$

Finally, the DC coefficient in each selected block is restored with the original $delta$ by

(13)$d=d^{\prime }+delta,$

where d represents the restored DC coefficient and d′ represents the DC coefficient in a visually transformed image. The aforementioned process can be illustrated with the examples in Figure 4.

Since one modified DC coefficient is 21 and the extracted $\frac{\left|delt{a}^{\prime }\right|}{4}$ is 4, we know the value of $\left|delt{a}^{\prime }\right|$ is 16. Since $\frac{\left|delt{a}^{\prime }\right|}{4}$ is even, delta should be non-negative. So, the value of delta is 16 and the original DCT coefficient can be recovered as 37 by adding it to 21. For another DC coefficient 23, the extracted data value is 5, which is an odd number, so delta should be negative according to Equation (12) and the value is −20. The original DCT coefficient is recovered by adding 23 to −20 (i.e., delta) to generate 3, which is the original DC coefficient value.

4. Experimental Results

In the experiments, test JPEG images were downloaded from the Images of Groups Dataset [25], which were collected from daily life. All experiments were implemented in the MATLAB environment. Three types of experiment were conducted to evaluate the proposed algorithm, including: (1) visual effects, quality of the transformed images and reversibility, (2) JPEG image file size and (3) security analysis.

4.1. Visual Effects, Image Quality and Reversibility

In this part, the proposed algorithm is evaluated in terms of the invisibility of the protected regions, quality of the unprotected blocks and reversibility. Four testing JPEG images used in the experiments and those with the selected regions protected are shown in Figure 5. In each test image, one or more specific regions were selected and the protected content was made visually invisible while the quality of the unprotected regions remained unchanged. The images recovered with the correct key were identical to the original ones, while the regions obtained with the wrong key were meaningless, as shown in Figure 6.

4.2. Change of JPEG Image Size

To evaluate the performance, the proposed algorithm was compared with Yuan’s algorithm [6], Li’s algorithm [7], P3 [11] and Scrambling JPEG [12] in terms of file size increment after visual transformation. In total one hundred JPEG images with different sizes were employed for testing. Table 1 shows the average JPEG file size increment with the five algorithms, respectively. The file size increment was computed by comparing the file size of a visually transformed image with the original one. The statistical results listed in Table 1 indicate that the proposed algorithm had the smallest increment in file size, which was 0.39% on average for 100 test images. In contrast, Yuan’s algorithm and P3 significantly increased the file size. It can be seen that the proposed algorithm was applied with a minimal file size increment of the JPEG images.

4.3. Security Analysis

The security of the proposed algorithm was analyzed by considering a brute force attack [22]. We conducted an experiment on one hundred JPEG images collected from the Images of Groups Dataset [25]. The block number of the protected regions in each JPEG image was calculated and is visualized in Figure 7. It can be seen that the block number ranged from 25 to 1600, while most of them were concentrated between 25 and 400. Besides, the experiments in [13] show that there are 26 nonzero AC coefficients in a block on average. Suppose that the attacker knows the location of the protected regions and the algorithm adopted for visual transformation. An exhaustive search of all the possible combinations of the permuted blocks and the flipped AC coefficients might be conducted. Given that there are at least 25 blocks in a protected region and 26 nonzero AC coefficients in a block, the number of combinations to obtain the permuted blocks is 25!, while there are 2^26×25 combinations to recover the flipped AC coefficients. Thus, the number of combinations to recover the original image is 25! × 2^26×25, which is close to 2⁷³⁴. It can be concluded that the proposed algorithm can resist brute force attack.

5. Concluding Remarks

In this paper, a reversible content and privacy protection algorithm has been proposed for JPEG images. Specifically, reversible data hiding and block scrambling have been combined to modify the DCT coefficients in the selected regions. Besides the invisibility of the protected content, reversibility of the transformation process has been achieved. The experimental results have shown that the proposed algorithm has good performance in terms of reversibility, invisibility of the user-selected region, quality of the transformed image and JPEG file size preservation. In addition, the proposed algorithm can resist a brute force attack against content and privacy protection in JPEG images.

Our future work is two-fold: (1) we will study how to improve the visual quality of a visually transformed JPEG image and (2) how to reduce the chance that a protected JPEG image attracts the suspicions of attackers will also be studied.

Author Contributions

Conceptualization, H.-T.W.; Data curation, X.C.; Formal analysis, Y.-m.C.; Funding acquisition, H.-T.W. and Y.-m.C.; Investigation, Y.H. and X.C.; Methodology, Y.H. and X.C.; Project administration, H.-T.W. and Y.-m.C.; Resources, Y.H. and X.C.; Software, Y.H. and X.C.; Writing—original draft, X.C.; Writing—review & editing, H.-T.W. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded in part by the Guangdong Province Key Area R&D Program of China with grant number 2019B010137004, in part by the Natural Science Foundation of China with grant numbers 61772208 and 61632013, in part by Hong Kong Baptist University under Grant RC-FNRA-IG/18-19/SCI/03 and RC-IRCMs/18-19/SCI/01, and in part by the Fundamental Research Funds for the Central Universities of China under Grant x2js-D2190700.

Conflicts of Interest

The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

Figures and Table

Figure 1. A flowchart of visual transformation for content and privacy protection in JPEG images.

Figure 2. Examples of modifying DC coefficients in selected blocks.

Figure 3. A flowchart of recovering the original JPEG image from the visually transformed one.

Figure 4. Examples of recovering DC coefficients in selected blocks.

Figure 5. Four JPEG images and the ones with user-selected regions protected.

Figure 6. JPEG images obtained from the visually transformed ones.

Figure 7. The histogram of the block number in the protected regions.

Table 1

Comparisons between five algorithms for content protection in JPEG images.

Algorithm Compared	JPEG File Size Increment (%)
Proposed algorithm	0.39
Yuan’s algorithm [6]	10.80
Li’s algorithm [7]	0.51
P3 [11]	7.5 ± 2.5
Scrambling JPEG [12]	2.04